Methods and apparatus to assess compliance of a virtual computing environment

ABSTRACT

Methods, apparatus, systems and articles of manufacture are disclosed to assess compliance of a virtual computing environment. An example method disclosed herein to assess compliance of computing resources of a computing environment includes monitoring for an occurrence of a change in a computing resource in the computing environment, and in response to detecting the occurrence and without waiting for batch testing, assessing compliance of the computing resource with a compliance policy.

RELATED APPLICATIONS

This patent arises from a non-provisional application which claims thebenefit of U.S. Provisional Application Ser. No. 61/819,462, filed onMay 3, 2013, which is hereby incorporated herein by reference in itsentirety.

FIELD OF THE DISCLOSURE

This disclosure relates generally to virtual computing, and, moreparticularly, to methods and apparatus to assess compliance of a virtualcomputing environment.

BACKGROUND

Compliance policies are often used to monitor and maintain resourcesdeployed in computing environments such as data centers. Policies suchas the Payment Card Industry Data Security Standard, the Department ofDefense Information Assurance Certification and Accreditation Process,vendor-supplied hardening guides, etc., provide strong guidelines forhandling and protecting sensitive data. Failure by an organization tocomply with these standards may result in loss of revenue, loss ofreputation, and/or loss of accreditation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an example virtual computing environment.

FIG. 2 is a block diagram of an example implementation of the examplevirtualization manager of FIG. 1.

FIG. 3 is a block diagram of an example implementation of the examplecompliance monitor of FIG. 2.

FIG. 4 is a flow chart representative of example machine-readableinstructions that may be executed to determine policy compliance ofcomputing resources of a virtual computing environment.

FIG. 5 is a block diagram of an example implementation of the examplecompliance assessor of FIG. 3.

FIGS. 6 and 7 are flow charts representative of example machine-readableinstructions that may be executed to perform event-driven compliancetesting of computing resources in a virtual computing environment.

FIGS. 8A and 8B illustrate an example compliance events record generatedby the example compliance monitor of FIGS. 2 and/or 3 to log compliancestates of a computing resource with different criteria of a compliancepolicy rule.

FIG. 9 is a block diagram of an example implementation of the examplecompliance measurer of FIG. 3.

FIGS. 10-14 are flow charts representative of example machine-readableinstructions that may be executed to measure compliance states ofcomputing resources in a virtual computing environment.

FIG. 15 is a block diagram of an example implementation of the exampleresults prioritizer of FIG. 3.

FIGS. 16 and 17 are flow charts representative of examplemachine-readable instructions that may be executed to identifypriorities of assessment results of policy compliance of computingresources in a virtual computing environment.

FIGS. 18-23 illustrate example graphical user interfaces generated bythe example reporter of FIG. 3 to generate reports for a user.

FIG. 24 is a block diagram of an example processing platform capable ofexecuting the example machine-readable instructions of FIGS. 4, 6, 7,10-14, 16 and 17 to implement the example compliance monitor of FIGS. 2and/or 3, the example compliance assessor of FIGS. 3 and/or 5, theexample compliance measurer of FIGS. 3 and/or 9, and/or the exampleresults prioritizer of FIGS. 3 and/or 15, respectively.

FIGS. 25A and 25B illustrate example truth tables representingfunctional values of an example policy.

FIG. 26 is a flow chart representative of example machine-readableinstructions that may be executed to measure compliance states ofcomputing resources in a virtual computing environment.

DETAILED DESCRIPTION

Physical computing environments include physical computing resourcessuch as servers, storage devices, etc. Physical computing resources maybe expensive to maintain and/or may require specialized knowledge tooperate and/or service. Virtual computing environments (sometimesreferred to as “virtual data centers”) virtualize such physicalresources or physical components making it possible for someone who doesnot actually own the physical computing resources (e.g., servers,storage components and networks) to utilize the resources throughcommercial transactions. Virtualizing aggregates and presents variousphysical resources as virtual resources in a virtual computingenvironment.

Over time, sensitive data and information has accumulated in virtualcomputing environments as virtual computing has been adopted in more andmore industries. For example, computing environments may store medicalrecords, credit card information, user identification information (e.g.,social security numbers), etc. Organizations are under strenuousrequirements to protect information stored within their computingenvironments. When combined with the trend towards virtualization, thegrowing inventory of virtual machines, and the emergence ofsoftware-defined data centers (e.g., virtual data centers), the task ofensuring compliance with various regulations and standards has becomedaunting. Further, given budget and other constraints, organizationsmust manage larger, more complex environments with the same or fewerinformation technology (IT) maintenance and management resources. Suchstretching of resources makes compliance an even greater challenge. Asthe quantity of stored data has exploded in the virtualization era,server configuration analysis has become time-consuming and oftenrequires multiple tools.

Compliance policies are used to increase the security and/or privacyprotections of computing environments. Such policies define rules forcomputing resource configurations (e.g., configuration settings) withinthe computing environments. Different compliance policies definedifferent sets and/or types of rules and/or requirements for deploymentand/or configuration settings of physical and/or virtual computingenvironments (e.g., physical and/or virtual data centers). Thus, acompliance policy describes the state and/or states that a computingenvironment and/or its resources should be in (e.g., a compliantposture) when the computing environment and/or its resources are incompliance with a policy. A compliance posture is a level of assurancethat adequate controls have been implemented to meet the goals of apolicy (e.g., information protection). A computing resource (or acomputing asset) is a type of resource with inspect-able attributesand/or properties. A configuration of the computing resource refers tothe state of the various attributes and/or properties. A configurationpolicy defines the configuration(s) acceptable for compliant operation.Computing resources include physical hosts, virtual machines, virtualnetwork configurations, data structures, data centers, softwareapplications, etc. The configuration settings of a computing resourcemay be subject to two or more policies having different purposes. Forexample, a computing resource may be subject to one or more policies topreserve information security, one or more policies to satisfy legal orregulatory requirements, one or more policies to satisfy corporate orindustry standards, one or more policies to maintain high performance orefficiency, one or more policies to satisfy contractual or quality ofservice goals, etc.

Different compliance policies (sometimes referred to as “policies,”“regulations,” “benchmarks,” “guidelines,” “standards” or“best-practices”) may originate from different policy writingorganizations such as the National Institute of Standards (NIST), theCenter for Internet Security (CIS), industry groups like the PaymentCard Industry (PCI), regulations like the Health Insurance Portabilityand Accountability Act (HIPAA), and/or product manufacturers. Eachpolicy represents a standard of operation with which a computingresource configuration must comply. Since policies originate withdifferent entities, the policies may be inconsistent with each other.For example, a first policy may specify that a system must rotate logfiles so that logs never exhaust disk space, while a second policy mayrequire that logs persist forever and that the system must ceaseoperation if log storage space becomes exhausted.

An assessment (sometimes referred to as a “compliance assessment,” a“policy assessment” or a “compliance test”) tests whether a resourceadheres to a designated set of policies. Conventionally, policies aremade available to system administrators as human-readable language(e.g., English) documents that put the onus on the system administratorto read, interpret and determine whether a computing environment is incompliance with the policy. In some examples, to reduce unnecessarybusiness risk, policy publishers and/or organizations may provide apolicy in a machine-readable form written using a structured language,program, or script. In such a format, the policy may contain one or morerules. Rules are Boolean-valued logical expressions (e.g., expressionsconstituting criteria formed by combining individual criterion usingoperators (or connectors) such as the logical AND operator, the logicalOR operator, or the logical NOT operator). Each criterion implementseither a Boolean-valued test of one resource configuration (e.g., aproperty) setting, or a reference by name to another criterion, set ofcriteria, or a rule. When executed in software, machine-readablepolicies automate the process of regulatory compliance.

When software is used to automate assessment (e.g., via OpenVulnerability Assessment Language (OVAL) or Extensible ConfigurationChecklist Description Format (XCCDF)), a management application managinga virtual computing environment can automatically test the virtualcomputing environment against numerous policies. When an acceptablecompliance posture is achieved, the management application attempts tomaintain the compliant posture. Thus, the management application musteffectively respond to changes in the environment (e.g., events) andinform a system administrator (or operator) of the impact of thechange(s).

FIG. 1 is an illustration of an example virtual computing environment100. The example virtual computing environment 100 of FIG. 1 includes anexample network of storage arrays 102 in communication with examplecomputing servers 104. The example network of storage arrays 102 may beimplemented using any suitable wired and/or wireless storage including,for example, one or more Fiber Channel Storage Area Network (SAN)arrays, one or more Internet Small Computer System Interface (iSCSI) SANarrays, one or more Network Attached Storage (NAS) arrays, etc. In theillustrated example, the network of storage arrays 102 are connected toand shared between groups of servers through storage area networks,thereby enabling aggregating storage resources and enabling increasedflexibility in provisioning the storage resources to, for example,example virtual machines 110.

In the illustrated example of FIG. 1, the example storage servers 104may be x86 servers in communication with the example network of storagearrays 102 via an example network 106. The network 106 of FIG. 1 may beimplemented using any suitable wired and/or wireless network(s) such as,for example, one or more data buses, one or more Local Area Networks(LANs), one or more wireless LANs, one or more cellular networks, theInternet, etc.

In the illustrated example of FIG. 1, the example computing servers 104provide example virtualization platforms 108. The example virtualizationplatforms 108 of FIG. 1 respectively execute on corresponding ones ofthe example computing servers 104. An example virtualization platform108 (sometimes referred to as a “virtualization layer,” a “hypervisor”or a “virtual machine monitor”) abstracts processor, memory, storageand/or other resources of the computing server 104 into one or morevirtual machines 110. In the illustrated examples, a virtual machine 110includes an operating system and/or executes one or more applications(sometimes referred to herein as “appliances”). In some examples, thevirtualization platform 108 may be installed on a computing server 104without an operating system (e.g., a bare metal hypervisor). In someexamples, the virtualization platform 108 may be installed on a storagedevice rather than on a computing server. The example virtualizationplatform 108 virtualizes and aggregates the underlying physical hardwareresources (e.g., the example network of storage arrays 102 and/or theexample computing servers 104) across the physical computing environmentand provides pools of virtual resources available for use in the virtualcomputing environment 100. Thus, by using the resources available fromthe physical components of the virtual computing environment 100, theexample virtual machines 110 may request resources dynamically as aworkload increases or release resources dynamically as the workloaddecreases.

The example virtual machines 110 of FIG. 1 may be designated to aparticular host, cluster or resource pool, or a data center when theyare created. A host is a virtual representation of computing and memoryresources of a physical computing server 104 executing a virtualizationplatform 108. When two or more physical computing servers 104 aregrouped to work and be managed as a whole (e.g., as a single entity orcomputing resource), the aggregate computing and memory resources may bereferred to as a cluster. In some examples, a computing server may bedynamically added or removed from a cluster. Computing and memoryresources from hosts and/or clusters may be partitioned into a hierarchyof resource pools.

To manage the virtual computing environment 100, the example virtualcomputing environment 100 of FIG. 1 includes an example virtualizationmanager 112. The example virtualization manager 112 provides a singlepoint of control to the virtual computing environment 100. In theillustrated example, the virtualization manager 112 manages theassignments of virtual machines 110 to be virtualized on correspondingones of the computing servers 104, and manages the assignments ofresources of the computing servers 104 to the virtual machines 110. Inthe illustrated example, the virtual computing environment 100 isaccessible via an example management client 114. For example, a virtualmachine 110 in the virtual computing environment 100 may be accessed viaa web access interface through a web browser of the client 114. In someother examples, the virtualization manager 112 may include one or moreinterfaces that enable other applications to manage the example virtualcomputing environment 100 and access the example virtualizationplatforms 108 and/or the example virtual machines 110.

FIG. 2 is a block diagram of an example implementation of the examplevirtualization manager 112 of FIG. 1. In the illustrated example of FIG.2, the virtualization manager 112 enables centralized management of theexample virtual computing environment 100. The example virtualizationmanager 112 aggregates physical resources via the example virtualizationplatforms 108 and presents a central collection of dynamic resourcesthat may be provisioned to example virtual machines 110 in the virtualcomputing environment 100. In the illustrated example, thevirtualization manager 112 includes an example user access controller202, an example distributed services controller 204, an example activedirectory interface 206, an example database interface 207, an examplevirtualization platform interface 208 and an example core servicescontroller 216.

In the illustrated example of FIG. 2, the user access controller 202enables creating and managing different levels of access to thevirtualization manager 112 based on, for example, different userclasses. User classes define different access rights to different users.For example, users in a first user class may be permitted to create andmanage physical virtualization hardware in the virtual computingenvironment 100, while users in a second user class may be permitted tomanage virtual resources within a particular resource pool in a virtualmachine cluster.

In the illustrated example of FIG. 2, the example distributed servicescontroller 204 enables extending the capabilities of a virtualizationplatform 108 beyond a single computing server 104. For example, thedistributed services controller 204 may include an example distributedresource scheduler (DRS) 220 to allocate and/or balance computingcapacity dynamically across collections of hardware resources forvirtual machines 110 (FIG. 1), an example availability manager 222 toenable restarting a virtual machine 110 on another computing server 104if, for example, an initial computing server hosting the virtual machine110 fails, an example fault tolerance manager 224 to enable creating asecondary copy of an original (or primary) virtual machine 110, anexample vMotion manager 226 to enable migration of virtual machine(s)from one computing server 104 to another computing server, etc.

In the illustrated example of FIG. 2, the virtualization manager 112includes example interfaces 206, 207, 208 to enable integratingadditional or third party applications with the virtualization manager112. For example, the active directory interface 206 of the illustratedexample enables communicating with an example active directory server210 to, for example, obtain user access control information. In theillustrated example, the database interface 207 enables thevirtualization manager 112 to communicate with an example database 212to store information such as virtual machine configurations, hostconfigurations, resources and virtual machine inventory, performancestatistics, events, alarms, user permissions and/or roles, etc. Theexample virtualization platform interface 208 of the illustrated exampleenables the virtualization manager 112 to communicate with one or moreapplication plug-ins 214, third party applications 215 and/or managementclients (e.g., the example management client 114 of FIG. 1).

In the illustrated example of FIG. 2, the virtualization manager 112includes the example core services controller 216 to provide managementservices for the virtual computing environment 100. For example, thecore services controller 216 may include an example virtual machineprovisioner 228 (to manage automated provisioning of virtual machinesand their resources), an example host and virtual machine configurer 230(to enable configuring hosts and virtual machines in the virtualcomputing environment 100), an example resources and virtual machineinventory manager 232 (to organize and manage virtual machines andresources in the virtual computing environment 100), an examplestatistics logger 234 (to log and report performance and resource usagestatistics of computing resources such as virtual machines, hosts,storage devices, and/or clusters), an example alarms and events manager236 (to track and warn users about potential resource overuse or eventconditions), an example task scheduler 238 (to schedule actions to occurat a given time), etc.

In the illustrated example of FIG. 2, the core services controller 216includes an example compliance monitor 218 to monitor policy complianceof the virtual computing environment 100. Although the compliancemonitor 218 may be implemented as an application in the core servicescontroller 216 in FIG. 2, the example compliance monitor 218 mayalternatively be included as a plug-in (e.g., the example plug-in 214)and communicate with the example virtualization manager 112 via thevirtualization platform interface 208. In the illustrated example, thecompliance monitor 218 receives a notification when an event (e.g., aconfiguration change in one or more computing resources of the virtualcomputing environment 100) is detected. The compliance monitor 218 ofthe illustrated example assesses the impact of the event on thecompliance posture of one or more resources with respect to one or morepolicies that are enforced in the virtual computing environment 100.

While an example manner of implementing the example virtualizationmanager 112 of FIG. 1 is illustrated in FIG. 2, one or more of theelements, processes and/or devices illustrated in FIG. 2 may becombined, divided, re-arranged, omitted, eliminated and/or implementedin any other way. Further, the example user access controller 202, theexample distributed services controller 204, the example activedirectory interface 206, the example database interface 207, the examplevirtualization platform interface 208, the example active directoryserver 210, the example database 212, the example plug-in 214, theexample third party application 215, the example core servicescontroller 216, the example compliance monitor 218, the exampledistributed resource scheduler 220, the example availability manager222, the example fault tolerance manager 224, the example vMotionmanager 226, the example virtual machine provisioner 228, the examplehost and virtual machine configurer 230, the example resources andvirtual machine inventory manager 232, the example statistics logger234, the example alarms and events manager 236, the example taskscheduler 238 and/or, more generally, the example virtualization manager112 of FIG. 1 may be implemented by hardware, software, firmware and/orany combination of hardware, software and/or firmware. Thus, forexample, any of the example user access controller 202, the exampledistributed services controller 204, the example active directoryinterface 206, the example database interface 207, the examplevirtualization platform interface 208, the example active directoryserver 210, the example database 212, the example plug-in 214, theexample third party application 215, the example core servicescontroller 216, the example compliance monitor 218, the exampledistributed resource scheduler 220, the example availability manager222, the example fault tolerance manager 224, the example vMotionmanager 226, the example virtual machine provisioner 228, the examplehost and virtual machine configurer 230, the example resources andvirtual machine inventory manager 232, the example statistics logger234, the example alarms and events manager 236, the example taskscheduler 238 and/or, more generally, the example virtualization manager112 could be implemented by one or more analog or digital circuit(s),logic circuits, programmable processor(s), application specificintegrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s))and/or field programmable logic device(s) (FPLD(s)). When reading any ofthe apparatus or system claims of this patent to cover a purely softwareand/or firmware implementation, at least one of the example user accesscontroller 202, the example distributed services controller 204, theexample active directory interface 206, the example database interface207, the example virtualization platform interface 208, the exampleactive directory server 210, the example database 212, the exampleplug-in 214, the example third party application 215, the example coreservices controller 216, the example compliance monitor 218, the exampledistributed resource scheduler 220, the example availability manager222, the example fault tolerance manager 224, the example vMotionmanager 226, the example virtual machine provisioner 228, the examplehost and virtual machine configurer 230, the example resources andvirtual machine inventory manager 232, the example statistics logger234, the example alarms and events manager 236 and/or the example taskscheduler 238 is/are hereby expressly defined to include a tangiblecomputer readable storage device or storage disk such as a memory, adigital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc.storing the software and/or firmware. Further still, the examplevirtualization manager 112 of FIG. 1 may include one or more elements,processes and/or devices in addition to, or instead of, thoseillustrated in FIG. 2, and/or may include more than one of any or all ofthe illustrated elements, processes and devices.

FIG. 3 is a block diagram of an example implementation of the examplecompliance monitor 218 of the example virtualization manager 112 of FIG.2. The example compliance monitor 218 enables determining policycompliance states of the computing resources of the virtual computingenvironment 100 and identifying computing resources that are not incompliance with the policies. 1 n the illustrated example of FIG. 3, theexample compliance monitor 218 includes an example compliance assessor302, an example compliance measurer 304, an example results prioritizer306, an example compliance database 308 and an example reporter 318. Theexample policy database 310 stores the policy (or policies) enforced inthe virtual computing environment 100 of FIG. 1. A policy may be storedin the policy database 310 as a document (or other type of datastructure such as a list, a table, etc.) including Boolean-valued checksfor the policy.

The example compliance monitor 218 of FIG. 3 includes the examplecompliance assessor 302 to assess or determine policy compliance ofcomputing resources in real-time (or substantially near real-time) aftera configuration change occurs (e.g., when a change event is detected) inthe virtual computing environment 100. For example, the complianceassessor 302 may monitor an event stream that signals configurationchanges to computing resources in the virtual computing environment 100.When an event corresponding to a configuration change of a computingresource (e.g., the example network of storage arrays 102 (FIG. 1), theexample computing resources 104 (FIG. 1), the example virtualizationplatform 108 (FIG. 1), and/or the example virtual machines 110) isdetected, the example compliance assessor 302 identifies other computingresources that are affected by the event. For example, the complianceassessor 302 may use an inventory list of computing resources thatrelate to other computing resources. The compliance assessor 302 of theillustrated example also tests those computing resources againstcompliance policies enforced in the virtual computing environment 100(e.g., by automatically applying policy rules, checks, and/or tests tothe states of data structures, resources, etc.). In this manner, theexample compliance assessor 302 of FIG. 3 provides updated complianceresults (e.g., assessment results) in real-time or substantial real-time(e.g., less than ten seconds). In some examples, the compliance assessor302 pre-populates the policy database 310 with one or more policies,populates the policy database 310 with a specified policy (or policies),and/or updates the policy database 310 periodically (e.g., every 24hours, every Monday at 9:00 AM, etc.) and/or aperiodically (e.g., whenprompted to update) with updates to existing policies such as modifying,adding, or removing criteria. In some examples, the compliance assessor302 stores the assessment results and/or other corresponding informationin the example results database 312 of the compliance database 308.

The example compliance monitor 218 of FIG. 3 includes the examplecompliance measurer 304 to measure the extent to which a computingresource is adhering to compliance policies during operation. Bymeasuring compliance, the example compliance measurer 304 may use acompliance policy as a metric for measuring the configuration quality ofa computing resource. In addition, the example compliance measurer 304may use a detected event, categorized as a gain or loss of compliance,to determine a normalized compliance score. In some examples, combiningcompliance scores over time, over groups of computing resources (e.g., acluster of virtual machines 110) and/or with other policies enables theexample compliance measurer 304 to distinguish relevant events fromirrelevant events (e.g., configuration changes that increase or decreasea compliance score versus configuration changes that do not affect thecompliance score), to track compliance trends, to identify complianceissues that are of greater importance to the corresponding user based onhistorical data, etc. In some examples, the compliance measurer 304stores the scores and/or other corresponding information in the examplescores database 314 of the compliance database 308.

The example compliance monitor 218 of FIG. 3 includes the exampleresults prioritizer 306 to prioritize (e.g., flag, identify, etc.)results of compliance assessments based on historical performances oractions of an organization (e.g., a system administrator, an informationtechnology (IT) administrator, etc.). The example results prioritizer306 analyzes historical data to determine the importance of anassessment result to an organization based on past actions responding tothe same type(s) of assessment results. In the illustrated examples,historical data includes previous assessment results (e.g., asdetermined by the example compliance assessor 302), responses to theresults (e.g., repairing a defect), changes in compliance scores (e.g.,as calculated by the example compliance measurer 304), rates of changesto compliance scores (e.g., as calculated by the example compliancemeasurer 304), etc. For example, if a first computing resource fails aPayment Card Industry (PCI) test (e.g., generates a first defect) and asecond computing resource fails a Center for Internet Security (CIS)test (e.g., generates a second defect), the example results prioritizer306 analyzes historical data to determine whether historical responsetimes are quicker (e.g., reduce defects at a greater rate) to address(e.g., repair, remediate, etc.) computing resources that fail a PCI testor to address repairs to computing resources that fail a CIS test. Insome such examples, the results prioritizer 306 may prioritize the firstand second defects accordingly (e.g., the first defect flagged to beaddressed before the second defect) based on, for example, previousresponses or rates of responses to similar defects. In some examples,the results prioritizer 306 stores the priority order and/or othercorresponding information in the example priority order database 316 ofthe compliance database 308 for display to a user in, for example, aresults report.

The example compliance monitor 218 of FIG. 3 includes the examplereporter 318 to generate reports based on information stored in thecompliance database 308. For example, the reporter 318 of theillustrated example retrieves assessment results stored in the resultsdatabase 312 and generates a report identifying the assessment resultsfor a computing resource(s), a policy (or policies) that was/weretested, satisfied and/or failed, the virtual computing environment 100,etc. In some examples, the reporter 318 retrieves compliance scores fromthe scores database 314 and generates a report identifying thecompliance scores for a computing resource(s), a policy (or policies),the virtual computing environment 100, etc. In some examples, thereporter 318 retrieves rankings from the priority order database 316 andgenerate a report identifying the ranked order of the assessment resultsto facilitate correcting issues in an order consistent with pastpractices. The example reporter 318 of FIG. 3 may generate reports asdocuments for printout, as a graphical user interface for display via,for example, a monitor, etc.

While an example manner of implementing the example compliance monitor218 of FIG. 2 is illustrated in FIG. 3, one or more of the elements,processes and/or devices illustrated in FIG. 3 may be combined, divided,re-arranged, omitted, eliminated and/or implemented in any other way.Further, the example compliance assessor 302, the example compliancemeasurer 304, the example results prioritizer 306, the examplecompliance database 308, the example policy database 310, the exampleresults database 312, the example scores database 314, the examplepriority order database 316, the example reporter 318 and/or, moregenerally, the example compliance monitor 218 of FIG. 2 may beimplemented by hardware, software, firmware and/or any combination ofhardware, software and/or firmware. Thus, for example, any of theexample compliance assessor 302, the example compliance measurer 304,the example results prioritizer 306, the example compliance database308, the example policy database 310, the example results database 312,the example scores database 314, the example priority order database316, the example reporter 318 and/or, more generally, the examplecompliance monitor 218 could be implemented by one or more analog ordigital circuit(s), logic circuits, programmable processor(s),application specific integrated circuit(s) (ASIC(s)), programmable logicdevice(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)).When reading any of the apparatus or system claims of this patent tocover a purely software and/or firmware implementation, at least one ofthe example compliance assessor 302, the example compliance measurer304, the example results prioritizer 306, the example compliancedatabase 308, the example policy database 310, the example resultsdatabase 312, the example scores database 314, the example priorityorder database 316 and/or the example reporter 318 is/are herebyexpressly defined to include a tangible computer readable storage deviceor storage disk such as a memory, a digital versatile disk (DVD), acompact disk (CD), a Blu-ray disk, etc. storing the software and/orfirmware. Further still, the example compliance monitor 218 of FIG. 2may include one or more elements, processes and/or devices in additionto, or instead of, those illustrated in FIG. 3, and/or may include morethan one of any or all of the illustrated elements, processes anddevices.

A flowchart representative of example machine-readable instructions forimplementing the compliance monitor 218 of FIGS. 2 and/or 3 is shown inFIG. 4. In this example, the machine-readable instructions comprise aprogram for execution by a processor such as the processor 2412 shown inthe example processor platform 2400 discussed below in connection withFIG. 24. The program may be embodied in software stored on a tangiblecomputer readable storage medium such as a CD-ROM, a floppy disk, a harddrive, a digital versatile disk (DVD), a Blu-ray disk, or a memoryassociated with the processor 2412, but the entire program and/or partsthereof could alternatively be executed by a device other than theprocessor 2412 and/or embodied in firmware or dedicated hardware.Further, although the example program is described with reference to theflowchart illustrated in FIG. 4, many other methods of implementing theexample compliance monitor 218 may alternatively be used. For example,the order of execution of the blocks may be changed, and/or some of theblocks described may be changed, eliminated, or combined.

The program of FIG. 4 begins at block 402 when the example complianceassessor 302 (FIG. 3) assesses compliance of one or more computingresources (e.g., the example network of storage arrays 102 (FIG. 1), theexample computing servers 104 (FIG. 1), etc.) in the virtual computingenvironment 100 (FIG. 1) with a policy (or policies). For example, thecompliance assessor 302 may receive notification of a newly generatedevent, identify the computing resource(s) associated with the event,test the identified computing resource(s) for policy compliance, andstore the assessment results (e.g., test satisfied or test unsatisfied)in the results database 312 (FIG. 3) of the compliance database 308(FIG. 3). Example processes disclosed herein that may be used toimplement block 402 are described below in connection with FIGS. 5-7.

At block 404, the example compliance measurer 304 (FIG. 3) determinescompliance score(s) of the computing resource(s) based on compliancemeasurements of assessment results that report adherence of thecomputing resource(s) to the policy (or policies). For example, thecompliance measurer 304 may retrieve assessment results from the resultsdatabase 312, calculate a normalizing factor based on a policy,calculate compliance scores for the policy using the normalizing factor,and store the compliance scores in the scores database 314 (FIG. 3) ofthe compliance database 308. Example processes disclosed herein that maybe used to implement block 404 are described below in connection withFIGS. 8-14.

At block 406, the example results prioritizer 306 determines responsepriorities of assessment results indicative of compliance failure states(e.g., defects) based on past repair actions and compliance scores. Forexample, the results prioritizer 306 may group defects into defectclasses based on the policy that generated the defect, processhistorical data associated with the defect classes, and estimate aranking (e.g., a priority order) for responding to the defect classes(e.g., analyzing defects, fixing defects, etc.) based on an analysis ofthe historical data, and store the response rankings in the priorityorder database 316 (FIG. 3) of the compliance database 308. Examplemethods disclosed herein that may be used to implement bock 406 aredescribed below in connection with FIGS. 15-17.

At block 408, the example reporter 318 (FIG. 3) generates a report basedon the compliance policy assessment. For example, the reporter 318 mayretrieve assessment results from the results database 312, compliancescores from the scores database 314 and/or response rankings from thepriority order database 316, and generate a report. In some examples,the report may be presented used a graphical user interface for userreview.

At block 410, the example compliance monitor 218 (FIG. 2) determineswhether to continue monitoring compliance of the virtual computingenvironment 100. If the compliance monitor 218 determines to continuemonitoring compliance of the virtual computing environment 100 (e.g.,while in communication with the example virtualization manager 112 (FIG.1)), control returns to block 402 to assess compliance of computingresource(s) of the virtual computing environment 100. Otherwise, if, atblock 410, the compliance monitor 218 determines not to continuemonitoring compliance (e.g., due to an application/process shutdownevent, a computer shutdown event, etc.), the example process of FIG. 4ends.

Example methods and apparatus to implement blocks 402, 404, and 406 ofFIG. 4 are disclosed herein. For example, block 402 may be implementedusing example methods and apparatus disclosed herein to performevent-driven compliance assessments of a virtual computing environment,as discussed below in connection with FIGS. 5-7. Block 404 may beimplemented using example methods and apparatus disclosed herein tomeasure compliance of a computing resource and, thus, the affect anevent has on policy compliance, as discussed below in connection withFIGS. 8-14. Block 406 may be implemented using example methods andapparatus disclosed herein to identify priorities when responding toassessment results, as discussed below in connection with FIGS. 15-17.

Although blocks 402, 404 and 406 are shown as occurring in the sameprocess flow in FIG. 4, in other examples, any one or more of the blocks402, 404, and 406 may be implemented independent of the others. Forexample, block 402 may be implemented to assess compliance of computingresources without implementing block 404 to determine compliance scoresas disclosed herein and/or without implementing block 406 to determineresponse priorities as disclosed herein. In some examples, block 404 maybe implemented to determine compliance scores without implementing block402 to assess compliance of computing resources as disclosed hereinand/or block 406 to determine response priorities as disclosed herein.In some examples, block 406 may be implemented to determine responsepriorities without implementing block 402 to assess compliance ofcomputing resources as disclosed herein and/or block 404 to determinecompliance scores as disclosed herein.

I. Assessing Compliance

Achieving and maintaining policy-based compliance presents manychallenges within enterprises and organizations subject to Governanceand Regulatory Compliance. For example, the number of policiesmaintained at an organization for their virtual computing environmentcontributes to a growing library of configuration tests. Due to thenumber of configuration tests and complexities associated therewith,organizations may wish to use a management application within theirvirtual computing environment to automate compliance assessment.

Prior management applications use a polling method to maintain anacceptable compliance posture. For example, on a pre-determined basis(e.g., every twenty-four hours), prior management applications maycollect (e.g., poll) the state of specified computing resources from avirtual computing environment as determined by a policy (or policies)and store the collected information in a centralized repository. Then,the prior management applications determine the compliance posture ofthe virtual computing environment in a batch after all the informationis collected and stored. Delays between polling may result in missinginterim states that were out of compliance during the interval beforepolling but changed to a compliant state through happenstance prior tothe polling. In prior systems, due to the amount of time needed tocollect, store and evaluate the collected information, notifications ofcompliance violations are not available for some time (e.g., hours)after a change occurs in the virtual computing environment. This processof “batch testing,” while sometimes effective, necessarily createsdelays between compliance failures and notifications to the IT staff,requires specialized training in the setup and maintenance of thecollecting software, and requires a carefully orchestrated pollinginterval. For example, an overly long polling interval increases therisk of missing configuration changes that may affect compliance. Forexample, a computing resource may change from an in-compliance state(e.g., online) to an out-of-compliance state (e.g., offline), and thenback into an in-compliance state (e.g., online) during a pollinginterval. As a result, the management application monitoring complianceof the virtual computing environment may never recognize that some eventcaused a computing resource to fall out-of-compliance, and thus, cannottake steps to prevent that event from re-occurring. In contrast, havinga polling interval that is too short risks decreasing performance of thevirtual computing environment due to the large amount of computation anddata collection required for batch testing. For example, processingresources and storage resources must be used to collect, store andevaluate the collected information each time the polling interval endsand a batch testing is performed.

Unlike prior systems that perform batch testing, example methods andapparatus disclosed herein use event-driven assessments, which provideupdated compliance results in real-time (or substantially nearreal-time). In addition, rather than batch testing potentially allcomputing resources in the virtual computing environment, examplesdisclosed herein enable limiting testing to those computing resourcesthat are affected by a configuration change (e.g., an event) on anoccurrence by occurrence basis, thereby resulting in improvedperformance of the virtual computing environment by leaving processingresources and storage resources free to execute other operations. As aresult, unlike prior systems, which must wait until system-wide batchtesting is performed, examples disclosed herein enable determiningcompliance of a computing resource (e.g., a portion of a system)immediately upon the occurrence of a configuration change affecting thecomputing resource (e.g., affecting a portion of the system) and/or atleast closer in time to the occurrence of the change (e.g., in a rangefrom zero to ten seconds (e.g., three seconds) of detecting an eventindicative of a computing resource configuration change) without waitingfor batch testing to be performed. That is, whereas batch processing istriggered by a timer and/or a threshold (e.g., a buffer is X percentfull), compliance processing in the illustrated example is driven bydetection of a computing configuration change. Thus, unlike batchprocessing, which depends on an external measure to trigger analysis, inexamples disclosed herein, compliance processing is driven by organicconfiguration events that suggest a compliance defect may have occurred.

FIG. 5 is a block diagram of an example implementation of the examplecompliance assessor 302 of the example compliance monitor 218 of FIGS. 2and 3. In the illustrated example of FIG. 5, the example complianceassessor 302 includes an example inventory builder 502, an examplecompliance tester 504, an example scope tester 506, an example checktester 508, an example event monitor 510, an example resource identifier512, an example batch tester 514, an example timer 516 and an examplestaleness monitor 518.

In the illustrated example of FIG. 5, the example compliance assessor302 includes the example inventory builder 502 to query the virtualcomputing environment 100 (FIG. 1) and register the computing resources(e.g., the example network of storage arrays 102 (FIG. 1), the examplecomputing servers 104 (FIG. 1), etc.) within the virtual computingenvironment 100. In some examples, the inventory builder 502 queries thevirtual computing environment 100 upon installation into the virtualcomputing environment 100. For example, the compliance monitor 218(FIGS. 2 and 3), the example compliance assessor 302 and/or the exampleinventory builder 502 may be installed in the virtual computingenvironment 100 with the virtualization manager 112 (FIGS. 1 and 2) ormay be installed at a later time (e.g., via the example plug-in 214(FIG. 2) or the example third party application 215 (FIG. 2)).

In some examples, the inventory builder 502 registers the computingresources as an inventory list. As used herein, an inventory list is adynamic list of computing resources that relate to other computingresources. In some examples, the inventory list may be organized byinventory type. For example, selecting a cluster list (e.g., via a webaccess client) may return identities or indications of all clusters inthe virtual computing environment 100 as well as lists of all resourcetypes that relate to the cluster (or clusters) selected. Some exampleresource types include lists of virtual machines, databases, servers,networks, etc. In some examples, the inventory builder 502 retrieves aninventory of resources (e.g., an inventory list) from anotherapplication communicating with the example virtualization manager 112(FIG. 1). The inventory builder 502 of the illustrated example storesthe inventory list in a database such as the example compliance database308 of FIG. 3.

In the illustrated example of FIG. 5, the example compliance assessor302 includes the example compliance tester 504 to test computingresources against policies to generate compliance results. As describedabove, a policy is a collection of rules containing criteria that arelogical expressions of one or more Boolean-valued criterion. Eachcriterion is either a test of a computing resource configuration (e.g.,a property) setting (e.g., logging, enabled=“true”), or is a referenceby name to another criterion, criteria or rule. For example, whendefining a rule, a criterion of the rule may utilize by reference thesame criteria of a second rule.

In the illustrated example of FIG. 5, criteria of a rule include a scopecondition tested by a scope tester 506 and a check condition tested by acheck tester 508. The process of testing a computing resource forcompliance with a policy using the scope tester 506 and/or the checktester 508 is referred to as an assessment. Thus, when the examplecompliance tester 504 receives a message to test a computing resource(e.g., from the example inventory builder 502, the example resourceidentifier 514, etc.), the compliance tester 504 accesses the examplepolicy database 310 (FIG. 3) to retrieve a policy that includes one ormore policy rules (sometimes referred to herein as “rules”).

While a policy applies to an inventory of computing resources, a rulemay be applicable to a subset of the inventory of computing resources.Thus, when testing the computing resource against the policy, a rulescope condition checked by the scope tester 506 determines whether arule applies to the computing resource. For example, a rule may applyonly to a certain operating system (e.g., LINUX operating system, etc.).Thus, those computing resources that do not execute that operatingsystem fail the rule scope condition. In the illustrated example, whenthe scope tester 506 determines that a computing resource fails the rulescope condition, the rule corresponding to the rule scope condition isinapplicable to that computing resource, and the example compliancetester 504 continues assessment with the next rule.

Continuing the above example, the set of computing resources that dosatisfy the scope condition (e.g., that do execute the specified LINUXoperating system) may be referred to as being within the scope of therule. In the illustrated example, if a computing resource passes (orsatisfies) a rule scope condition performed by the scope tester 506, thecheck tester 508 determines whether the computing resource satisfies arespective rule check condition. In the illustrated example, a rulecheck condition may be used to compare a configuration setting (e.g., aproperty) of a computing resource to a value required by the rule checkcondition for that computing resource. If the configuration settingvalue of the computing resource satisfies the check condition, the checktester 508 determines that the computing resource passes (or satisfies)the rule check condition. However, if the computing resourceconfiguration setting value does not satisfy the rule check condition,then the check tester 508 determines that the computing resource fails(or does not satisfy) the rule check condition. When a computingresource fails a rule check condition, a defect is detected.

In the illustrated example of FIG. 5, the example check tester 508stores the result of the rule check test in a database such as theexample results database 312 of FIG. 3. In some examples, the checktester 508 appends a timestamp to the result when storing the result inthe results database 312. Thus, for example, a results entry in theresults database 312 may include a computing resource identifier(identifying the computing resource tested), a rule identifier(identifying the rule and the policy tested), a result of the policytest, the results of the scope tester 506 on each scope condition testedby the scope tester 506, the results of the check tester 508 on eachrule check condition tested by the check tester 508, the results of eachcriterion test, and a timestamp.

In the illustrated example of FIG. 5, the example compliance assessor302 includes the example event monitor 510 to detect newly generatedevents in the virtual computing environment 100. In some examples, theevent monitor 510 may receive messages from the virtualization manager112 (FIG. 1) when the state of a computing resource changes. Forexample, an application in the core services controller 216 (FIG. 2) maypublish a list (sometimes referred to as a “feed” or “stream” (e.g., anevents stream)) of configuration events that affect the virtualcomputing environment 100 (e.g., via a web feed such as an Atom feed, aRich Site Summary (RSS) feed, etc.). In some such examples, the eventmonitor 510 may subscribe to the feed.

In some examples, the event monitor 510 may use components of one ormore kernels included in the virtual computing environment 100 togenerate events. As discussed above in connection with the examplevirtual machines 110 (FIG. 1), each virtual machine 110 may include anoperating system having an operating system kernel. As a result, theevent monitor 510 may subscribe to a different operating system kernelfor each of the different operating systems operating on the virtualmachines 110. For example, the event monitor 510 may generate an eventbased on a change indication received from an Inotify subsystem or aFileSystem Watcher subsystem. Inotify is a Linux kernel subsystem thatnotifies subscribed applications of changes to a file system via, forexample, a web feed (or an events stream). Similarly, FileSystem Watcheris a .NET class subsystem that enables an application (e.g., the exampleevent monitor 510) to subscribe to file system events. Upon receiving anotification of a newly generated event via a web feed (or an eventsstream), the example event monitor 510 outputs a notification messageidentifying the event via, for example, an events stream.

In the illustrated example of FIG. 5, the compliance assessor 302includes the example resource identifier 512 to identify one or morecomputing resources associated with a detected event. For example, theresource identifier 512 of the illustrated example parses thenotification message retrieved from the example event monitor 510 toidentify one or more computing resources. In some examples, the resourceidentifier 510 may identify additional information regarding thecomputing resources such as the new state of the computing resources,other computing resources related to the event-associated computingresources (e.g., via the inventory list generated by the inventorybuilder 502), etc. In the illustrated example of FIG. 5, the resourceidentifier 512 provides the identified information (e.g., theevent-associated computing resources, compliance states of the newcomputing resources, other related computing resources, etc.) in a listto the compliance tester 504 to determine whether the event affectedpolicy compliance of other computing resources of the virtual computingenvironment 100. In this manner, the example compliance assessor 302assesses compliance of computing resources of the virtual computingenvironment 100 with a compliance policy (or policies) in response todetecting an event and without waiting for batch testing of thecomputing resources to be performed by, for example, the batch tester514.

In the illustrated example of FIG. 5, the compliance assessor 302includes the staleness monitor 518 to identify computing resource(s)that have not been assessed in a pre-determined amount of time (e.g.,twenty-four hours). For example, the staleness monitor 518 mayperiodically (e.g., every five minutes) check timestamps associated withresource assessments to determine if any computing resources have notbeen assessed within a threshold duration (e.g., in a pre-determinedamount of time) and, thus, have stale assessments (e.g., outdatedassessments or assessments that may no longer be valid). In someexamples, an assessment becomes stale when a subsequent assessment(e.g., an event-associated assessment, a batch test, etc.) has not beenperformed on the corresponding computing resource in a pre-determinedamount of time. In the illustrated example of FIG. 5, when the stalenessmonitor 518 determines an assessment to be stale, the staleness monitor518 provides an identity of the corresponding resource to the compliancetester 504 to test the resource against policies to generate a new(e.g., fresh) assessment result for the resource. In some such examples,the events monitor 519 may include an indication that thestaleness-related events are lower priority relative to theevent-associated events, thereby allowing the compliance tester 504 toperform event-related assessments first, and to performstaleness-related events when processing resources are available. Forexample, the staleness monitor 518 may check timestamps every fiveminutes and identify a resource with a stale assessment (e.g., anassessment performed more than twenty-four hours prior) having atimestamp indicating that it was performed on Jan. 1, 2013, at 12:05 pm.However, the compliance tester 504 may not have processing resourcesavailable to perform staleness-related assessments until 12:11 pm thatday. In such an example, the staleness monitor 518 may not identify theassessment performed the prior day at 12:11 pm to be stale until thestaleness monitor 518 performs its 12:15 pm check on Jan. 2, 2013. Inthis manner, the compliance assessor 302 maintains fresh complianceassessments (e.g., valid or updated compliance assessments) whileleaving processing resources and storage resources free to execute otheroperations.

In the illustrated example of FIG. 5, the example compliance assessor302 includes the example batch tester 514 to perform batch testing ofthe virtual computing environment 100. For example, when initiated, thebatch tester 514 may query the virtual computing environment 100, updatethe inventory list to include the computing resource states of thecomputing resources at that time, and then test each of the computingresources in the virtual computing environment 100 against a policy (orpolicies) enforced in the virtual computing environment 100.

In the illustrated example of FIG. 5, the example compliance assessor302 includes the example timer 516 to maintain a regular pollinginterval between batch tests. For example, the timer 516 may initiatethe batch tester 514 periodically (e.g., every thirty minutes) to batchtest the virtual computing environment 100. In some examples, thepolling interval may be dynamic and vary based on, for example, aworkload. In this manner, the example compliance assessor 302 mayperform event-driven assessments (e.g., via the event monitor 510) andregular (e.g., periodic) batch tests (e.g., via the batch tester 514).

In some examples, the batch tester 514 may be included with the eventmonitor 510. In some examples, the batch tester 514 may be triggered bythe event monitor 510. The example event monitor 510 detects newlygenerated events in the virtual computing environment, which enablesreal-time (or substantially near real-time) compliance assessment. Theexample batch tester 514 performs batch tests of each of the computingresources of the example virtual computing environment 100, includingany computing resources that were recently tested as a result of adetected occurrence of a configuration change, at regular pollingintervals. Thus, the combination enables the collected data set to bemore robust and complement each other. For example, the complianceassessor 302 can perform compliance tests that utilize data indicated byevents detected by the event monitor 510 and data collected at intervalsby the batch tester 514. In some examples, the compliance assessor 302performs compliance tests that utilize data indicated by events detectedby the example event monitor 510 and related data identified by theexample inventory builder 502 that is unchanged from the previous batchtest.

While an example manner of implementing the compliance assessor 302 ofFIG. 3 is illustrated in FIG. 5, one or more of the elements, processesand/or devices illustrated in FIG. 5 may be combined, divided,re-arranged, omitted, eliminated and/or implemented in any other way.Further, the example inventory builder 502, the example compliancetester 504, the example scope tester 506, the example check tester 508,the example event monitor 510, the example resource identifier 512, theexample batch tester 514, the example timer 516, the example stalenessmonitor 518 and/or, more generally, the example compliance assessor 302of FIG. 3 may be implemented by hardware, software, firmware and/or anycombination of hardware, software and/or firmware. Thus, for example,any of the example inventory builder 502, the example compliance tester504, the example scope tester 506, the example check tester 508, theexample event monitor 510, the example resource identifier 512, theexample batch tester 514, the example timer 516, the example stalenessmonitor 518 and/or, more generally, the example compliance assessor 302could be implemented by one or more analog or digital circuit(s), logiccircuits, programmable processor(s), application specific integratedcircuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or fieldprogrammable logic device(s) (FPLD(s)). When reading any of theapparatus or system claims of this patent to cover a purely softwareand/or firmware implementation, at least one of the example inventorybuilder 502, the example compliance tester 504, the example scope tester506, the example check tester 508, the example event monitor 510, theexample resource identifier 512, the example batch tester 514, theexample timer 516 and/or the example staleness monitor 518 is/are herebyexpressly defined to include a tangible computer readable storage deviceor storage disk such as a memory, a digital versatile disk (DVD), acompact disk (CD), a Blu-ray disk, etc. storing the software and/orfirmware. Further still, the example compliance assessor 302 of FIG. 3may include one or more elements, processes and/or devices in additionto, or instead of, those illustrated in FIG. 5, and/or may include morethan one of any or all of the illustrated elements, processes anddevices.

Flowcharts representative of example machine-readable instructions forimplementing the compliance assessor 302 of FIGS. 3 and/or 5 are shownin FIGS. 6 and 7. In these examples, the machine-readable instructionscomprise a program(s) for execution by a processor such as the processor2412 shown in the example processor platform 2400 discussed below inconnection with FIG. 24. The program may be embodied in software storedon a tangible computer readable storage medium such as a CD-ROM, afloppy disk, a hard drive, a digital versatile disk (DVD), a Blu-raydisk, or a memory associated with the processor 2412, but the entireprogram and/or parts thereof could alternatively be executed by a deviceother than the processor 2412 and/or embodied in firmware or dedicatedhardware. Further, although the example program is described withreference to the flowcharts illustrated in FIGS. 6 and/or 7, many othermethods of implementing the example compliance assessor 302 mayalternatively be used. For example, the order of execution of the blocksmay be changed, and/or some of the blocks described may be changed,eliminated, or combined. Although the operations in the flow diagrams ofFIGS. 6 and 7 are shown in seriatim, the examples of FIGS. 6 and 7 mayadditionally or alternatively be implemented so that one or more of theoperations are performed in parallel with one or more other ones of theoperations.

The program of FIG. 6 begins at block 602 when the example event monitor510 (FIG. 5) determines whether a computing resource in the examplevirtual computing environment 100 (FIG. 1) generated an event. Forexample, the event monitor 510 may subscribe to a feed from anapplication in the example core services controller 216 (FIG. 2) andreceive notifications of newly generated events via an event stream. If,at block 602, the event monitor 510 does receive notification of a newlygenerated event, then, at block 604, the example resource identifier 512(FIG. 5) identifies a computing resource(s) (e.g., the example computingserver 104 (FIG. 1)) associated with the event. For example, theresource identifier 512 may parse information in a message from theevent monitor 510 to identify one or more computing resources associatedwith the event. In some examples, the resource identifier 512 queriesthe inventory builder 502 (FIG. 5) to identify computing resourcesrelated to the event-associated computing resources. In some examples,the resource identifier 512 queries the policy database 310 (FIG. 3) toretrieve a policy (or policies) enforced in the virtual computingenvironment 100.

At block 606, the example compliance tester 504 (FIG. 5) tests the oneor more event-associated computing resources (and, if applicable,computing resources related to the event-associated computing resources)against a policy identified by the resource identifier 512, as describedin greater detail below in connection with FIG. 7. For example, thecompliance tester 504 may compare a computing resource state against arule check condition of a policy rule and store the result of thecomparison in the example results database 312 (FIG. 3). At block 608,the compliance assessor 302 determines whether to continue performingcompliance assessments. For example, the compliance assessor 302 (FIGS.3 and/or 5) may stop performing compliance assessments during a poweroutage. In some such examples, the example process of FIG. 6 ends.Otherwise control returns to block 602 to determine whether a computingresource in the example virtual computing environment 100 generated anevent.

Returning to block 602, if the event monitor 510 does not receivenotification of a newly generated event, control advances to block 610,at which the example staleness monitor 518 (FIG. 5) determines whetherthe virtual computing environment 100 includes computing resource(s)that have not been assessed within a threshold duration (e.g.,twenty-four hours). For example, the staleness monitor 518 may parsethrough timestamps associated with past assessments from the exampleresults database 312 and determine any computing resources withassessments older than twenty-four hours. If, at block 610, thestaleness monitor 518 identifies stale assessments, then, at block 612,the example staleness monitor 518 identifies the computing resource(s)corresponding to the stale assessments. For example, the stalenessmonitor 518 includes identifiers of the identified resources in a listto the compliance tester 504. In some examples, the staleness monitor518 may include a priority marker with the identified resourcesindicating that performing assessments of the identified resources(e.g., staleness-related assessments) is of lower priority thanperforming assessments of event-associated computing resources. Controlthen proceeds to block 606 to test compliance of the computing resourcesagainst the policy (or policies) enforced in the virtual computingenvironment 100.

Returning to block 610, if the staleness monitor 518 does not determineany stale assessments, control advances to block 614, at which theexample compliance assessor 302 determines whether to perform batchtesting of the virtual computing environment 100. For example, the timer516 (FIG. 5) may expire and initiate batch testing processing.Otherwise, the compliance assessor 302 may be event-driven and initiatecompliance assessments of the virtual computing environment 100 when anevent in the virtual computing environment 100 is generated (ordetected).

If, at block 614, the compliance assessor 302 determines not to performbatch testing (e.g., the timer 516 does not indicate that the pollinginterval expired), then, control returns to block 602. Otherwise controladvances to block 616, at which the batch tester 514 (FIG. 5) collectscomputing resource states for all computing resources included in thevirtual computing environment 100. For example, the batch tester 514 mayquery the virtual computing environment 100 to retrieve/identify themost recent computing resource states of the virtual computingenvironment 100 computing resources. Thus, in some examples, the batchtester 514 may query hundreds of computing resources to collect thecomputing resource states of the computing resources. At block 618, thebatch tester 514 batches the computing resource states for the virtualcomputing environment 100 computing resources. For example, the batchtester 514 may update the inventory list (stored in the examplecompliance database 308 (FIG. 3)) to include the most recent computingresource states of the computing resources. As discussed above,collecting computing resource states and storing the computing resourcestates is costly in terms of processing resources and storage resources,thereby resulting in inefficient compliance assessments. Control thenproceeds to block 606 to test compliance of the computing resourcesagainst the policy (or policies) enforced in the virtual computingenvironment 100.

The program of FIG. 7 illustrates an example method of testing whether acomputing resource is in compliance with a policy enforced in thevirtual computing environment 100 (FIG. 1). The example program of FIG.7 may be used to implement block 606 of FIG. 6. The program of FIG. 7begins at block 702 when the example compliance tester 504 (FIG. 5)selects a computing resource to test. For example, the compliance tester504 may select a computing resource from the list of identifiedcomputing resources, as described above in connection with the resourceidentifier 512 (FIG. 5). At block 704, the compliance tester 504retrieves a policy including one or more rules for testing against thecomputing resource. For example, the compliance tester 504 may retrievethe policy from the policy database 310 (FIG. 3). At block 706, thecompliance tester 504 selects one of the one or more rules to test. Atblock 708, the compliance tester 504 determines whether the computingresource is within the scope of the rule. For example, the scope tester506 (FIG. 5) tests the computing resource against the scope condition ofthe rule. If, at block 708, the scope tester 506 determines that thecomputing resource fails the scope condition of the rule, controlproceeds to block 716. Otherwise if the computing resource satisfies thescope condition of the rule, control advances to block 710, at which thecompliance tester 504 determines whether the computing resourcesatisfies the rule. For example, the check tester 508 (FIG. 5) tests thecomputing resource state against the check state of the rule. When thecomputing resource state is the same as (e.g., matches) the check stateof the rule, the computing resource passes the rule check condition and,at block 712, the check tester 508 logs that the computing resourcesatisfies the rule in the example results database 312 of FIG. 3. Whenthe computing resource does not satisfy (e.g., does not match) the rulecheck condition, at block 714, the check tester 508 generates a defect(e.g., a defect result) and the check tester 508 logs the defect in theresults database 312.

After the check tester 508 logs the assessment result in the resultsdatabase 312 at block 712 or block 714, or if the computing resourcefailed the rule scope condition at block 708, at block 716, thecompliance tester 504 determine(s) whether the policy includes anotherrule to test. For example, the compliance tester 504 may determinewhether the tested rule was the last rule in the policy. If, at block716, the compliance tester 504 determines the policy includes anotherrule to test, control returns to block 706 to test a new rule againstthe computing resource.

Otherwise, if, at block 716, the compliance tester 504 determines thatthe last rule in the policy was tested, then, at block 718, thecompliance tester 504 determines whether another policy is enforced onthe computing resource. For example, the compliance tester 504 may querythe policy database 312 for another policy. If the policy database 512indicates that there is another rule to test, then control returns toblock 704 to retrieve another policy from the policy database 312.Otherwise, if the policy database 312 indicates that there is notanother policy to test (e.g., the example policy database 312 returns anempty reply or a null value), then, at block 720, the compliance tester504 determines whether there is another computing resource to test. Forexample, the compliance tester 504 may check if the list of computingresources provided by the resource identifier 514 includes anothercomputing resource. If, at block 720, the compliance tester 504determines that there is another computing resource to test (e.g., thecomputing resources list includes a non-tested computing resource), thencontrol returns to block 702 to select another computing resource totest. Otherwise, if the program of FIG. 6 is initiated by a newlygenerated event or if the compliance tester 504 determines that there isnot another computing resource to test at block 720, then controlreturns to a calling function or process such as the example program ofFIG. 6, and the example process of FIG. 7 ends.

II. Measuring Compliance

After a configuration change (or event) is received and compliance of acorresponding computing resource(s) is determined, examples disclosedherein may be used to measure the extent to which the computingresource(s) are adhering to a compliance policy. Such disclosed examplesmay be useful to a system administrator managing the virtual computingenvironment 100 (FIG. 1). For example, if a computing resource fails apolicy, the system administrator may use examples disclosed herein toassess the degree to which the computing resource is in compliance(e.g., an in-compliance amount or a degree of compliance with a policy).Some disclosed examples may be used to identify defects for whichquicker responses (e.g., corrections) would be relatively morebeneficial to the virtual computing environment. In addition, the systemadministrator may use examples disclosed herein to measure complianceover time, over a group of computing resources and/or across acollection of policies.

FIGS. 8A and 8B illustrate an example compliance events record 800indicative of compliance states of a computing resource with respect todifferent criteria of a policy (P) (e.g., OVAL definition CCE-14816-3 oroval:nist.usgcb.rhel:def;20159). The compliance events record 800 of theillustrated example includes results of eight criteria of the policy(P). In the illustrated example, the compliance events record 800includes compliance failures 802A, 802B, 802C indicating that thecomputing resource failed three of the eight criteria (e.g., marked withan “X” inside a circle). The compliance events record 800 also includesfive compliance successes 804A, 804B, 804C, 804D, 804E indicating thatthe computing resource passed five of the criteria (e.g., marked with acheck mark).

Some prior compliance measurement tools measure compliance of a policyas a whole. That is, some prior compliance measurement tools provideonly a pass (or satisfied or “1”) indicator or fail (or unsatisfied or“0”) indicator of the computing resource with respect to the policy as awhole (e.g., policy compliance failure 806). In such prior compliancemeasurement tools, a system administrator is only informed when thecompliance of a computing resource with a policy as a whole unit changesfrom compliance success to compliance failure or from compliance failureto compliance success. Thus, the system administrator is unaware when anindividual criterion of the policy changes compliance states. Forexample, if one of the five compliance successes 804 changes to acompliance failure 802, and the compliance status corresponding to thewhole, unitary policy continues to display an indicator of only thepolicy compliance failure 806, the system administrator will be awarethat the compliance of the corresponding computing resource with thepolicy continues to be in a fail state, but the system administratorwill not be able to discern, based on the compliance failure indicator806, that the computing resource moved further away from a complianceposture (e.g., the compliance successes of the computing resourcedecreased from five compliance successes 804 to four compliancesuccesses 804).

Compliance may be measured using numerical values. Some prior compliancemeasurement tools measure compliance based on a raw score of, forexample, compliance successes versus the total tested policy criteria.For example, prior compliance measurement tools may calculate a score of“5/8” (e.g., 62.5 percent compliance for the five compliance successes804 of the eight total tested policy criteria of FIGS. 8A and 8B) basedon the compliance events record 800 of FIGS. 8A and 8B. While this maybe true as a raw score, it lacks relevance to whether the computingresource is in or out of compliance. For example, passing all eight ofthe criteria noted in the compliance events record 800 may not benecessary to be in compliance with the policy (P). In the example ofFIGS. 8A and 8B, the compliance failure 802C causes the policycompliance failure 806, while the compliance failures 802A, 802B are notrelevant due to the compliance successes 804A, 804B. As discussed abovein connection with compliance policies, a policy may be expressed as alogical combination of one or more criteria. Equation 1 below is anexample logical representation of an example policy (P) illustrated inFIGS. 8A and 8B.

P=(¬V ₁ ∥V ₂)&&(¬V ₃ ∥V ₄)&&V ₅&&V ₆&&V ₇&&V ₈  Equation 1:

In Equation 1 above, the policy (P) is a logical combination ofcriteria. Each criterion is a Boolean-valued check (e.g. V₂) or logicalnegation of a check (e.g. ¬V₁). In Equation 1, “∥” is the logical ORoperator, and “&&” is the logical AND operator. Example Equation 1represents a policy compliance test that defines a threshold for policycompliance (e.g., a compliance posture). Thus, using Equation 1 above inconnection with the compliance events record 800 of FIGS. 8A and 8B, acomputing resource tested against the policy (P) need only satisfy sixof the eight criteria to pass. For example, the computing resource mustonly comply with criterion (¬V₁) or comply with criterion (V₂), mustcomply with criterion (¬V₃) or comply with criterion (V₄), and mustcomply with each of criterion (V₅, V₆, V₇, and V₅). However, while priorcompliance measurement tools that use a raw score of “6/8” may determinethat the computing resource fails the policy (P), the raw score of “6/8”does not agree with such a compliance failure. Furthermore, in someexamples, a computing resource may be in compliance with a policy andnever have a raw score of “8/8.” For example, the check (V₁) of thepolicy (P) may determine whether the computing resource is operating a32-bit architecture while the check (V₃) of the policy (P) may determinewhether the computing resource is operating a 64-bit architecture. Insome such examples, while the computing resource may comply with thepolicy (P) by operating either the 32-bit architecture or the 64-bitarchitecture, the computing resource does not operate both architecturessimultaneously (e.g., at the same time or substantially near the sametime) and, thus, cannot achieve a raw score of “8/8.”

Accordingly, example methods and apparatus disclosed herein enablemeasuring compliance based on the compliance state of individualcriterion of a policy. For example, as explained in connection withexample Equation 1 above the example compliance events record 800 ofFIGS. 8A and 8B, examples disclosed herein process the properties of thepolicy (P) so that if a logically OR'ed criterion (e.g., the criterion(¬V₃)) is satisfied, the first (satisfied) logically OR'ed criterion(¬V₃) is processed as a necessary criterion and the second logicallyOR'ed criterion (e.g., the criterion (V₄)) is processed as anunnecessary criterion. In some such examples, changes to the compliancestate of the second logically OR'ed criterion (V₄) are treated asirrelevant in measuring compliance of the computing resource. That is,when the first logically OR'ed criterion (¬V₃) is satisfied (e.g.,marked by a compliance success 804B), a change in the compliance stateof the second logically OR'ed criterion (V₄) is irrelevant indetermining whether the computing resource is in compliance with thepolicy (P). Thus, unlike prior systems, examples disclosed herein enablerecognizing necessary criteria and/or unnecessary criteria so that acompliance score increases as necessary criteria are satisfied,decreases as necessary criteria become unsatisfied and is unaffected byconfiguration changes that are neutral (e.g., unnecessary criteriachange values and/or no-longer necessary criteria change values).

Compliance scores from compliance measurements of different policies maynot always be meaningfully comparable to one another because policiesoften originate from different sources (e.g., different publishers) thatuse different policy criteria or compliance success standards. Forexample, a computing resource may be required to comply with a firstpolicy including four criteria provided by a policy publishing source,and may be subject to also comply with a second policy including onehundred criteria provided by the same or different policy publishingsource. Thus, each policy effectively provides a different level ofdifficulty for achieving compliance and, thus, a different compliancemetric for measuring compliance.

Examples disclosed herein enable calculating a normalized (orstandardized) compliance score for each policy based on the propertiesof the policy. In this manner, normalized compliance scores acrossdifferent policies are meaningfully comparable to one another. Theproperties of a policy include the number of criteria (V_(m)) in thepolicy, the relationship between criteria (e.g., the logicalrepresentation of the policy including the logical AND operator, thelogical OR operator, the logical NOT operator, etc.), and the compliancestate (e.g., compliance failure or compliance success) of each of thecriteria (V_(m)). Examples disclosed herein use the properties of thepolicy to calculate a surprisal value of the policy, which may then beused to calculate a normalized compliance score for a computing resourceagainst the policy. The surprisal value (or uncertainty) is definedherein as the amount of information expressed by a particular outcome,given a distribution of outcomes and their probabilities.

To illustrate, consider a dice-rolling policy (P_(dice)) where two dice(d₁, d₂) are rolled and the policy is satisfied when the sum of the tworolled dice is to equal eleven. Equation 2 below is a logicalrepresentation of the policy (P_(dice)).

P _(dice)=(d ₁=5&& d ₂=6)∥(d ₁=6&& d ₂=5)  Equation 2:

In Equation 2 above, the surprisal value for the policy (P_(dice)) maybe calculated based on the number of combinations of two dice rolls thatsum to eleven. Thus, the surprisal value corresponds to the amount ofuncertainty that something will happen given the properties of thesystem. For example, in Equation 2 above, the surprisal value depends onthe number of criteria (e.g., two dice rolled), the logical relationshipbetween the criteria (e.g., the sum of the two dice), and the values ofthe two dice. Thus, in Equation 2 above, the surprisal value isdifferent when the result of no dice is known, when the result of one ofthe two dice is known, and when the result of both dice are known. Thatis, the amount of uncertainty in satisfying the policy (P_(dice))decreases when the result of one die is known, and decreases again(e.g., zero uncertainty) when the results of the two dice are known. Thesurprisal value calculated before any information is known may bereferred to as the maximum surprisal (or uncertainty) of the policy(P_(dice)). The surprisal value when one of the dice values is known maybe referred to as the remaining surprisal value of the policy (P_(dice))given the amount of information known (e.g., the first die result). Thedifference in values between the maximum surprisal value and theremaining surprisal value after information is known (e.g., the firstdie result) corresponds to the amount of surprisal lost and, thus, maybe referred to as information gain. The ratio of the information gain tothe maximum surprisal value may be referred to as the compliance scorefor the policy (P_(dice)) given the information known.

Thus, examples disclosed herein calculate a compliance score using theproperties of the policy. As a result, examples disclosed herein enablecomparing and/or combining compliance scores with one another. Forexample, the compliance score can be combined over time and/or over agroup of computing resources that are measured against the same policybecause the compliance score is with respect to the same maximumsurprisal. In addition, compliance scores over policies may be combinedby using the ratio of the maximum surprisal of each policy. As a result,the compliance posture of a computing resource may be evaluated based onthe combined measured compliance of a group of computing resources, thecombined measured compliance of a computing resource over time and/orthe combined measured compliance of the computing resource against twoor more policies.

Moreover, when calculating a compliance score, examples disclosed hereindo not depend on the type of policy (e.g., a vulnerability policy, acompliance policy, etc.) being enforced. In addition, some examplesdisclosed herein may be used to calculate a compliance score for apolicy that does not enable automated compliance assessments (e.g., thepolicy rules are not machine-readable), but that includessatisfied/unsatisfied (e.g., pass/fail, true/false, etc.) truth-valuesto perform a manual compliance assessment. For example, a compliancescore may still be calculated for a policy encoded in the Open ChecklistInteractive Language (OCIL) where the results are human answers to aquestionnaire. That is, examples disclosed herein may calculate acompliance score for any policy that can be represented as a Booleanexpression. For example, examples disclosed herein may calculate acompliance score for reports generated by methods that did not useexample scoring processes disclosed herein (e.g., a different scoringprocess, a legacy scoring process, etc.). Thus, in some examples,examples disclosed herein retroactively calculate a compliance score fora legacy scoring process. In some examples, compliance scores forreports are generated using a first compliance score calculated by afirst scoring technique (e.g., example scoring processes disclosedherein) and one or more other scoring techniques different from thefirst scoring technique.

Some prior compliance measurement systems generate compliance reportsbased on assessment results of computing resources of the virtualcomputing environment 100 (FIG. 1) against a policy. A compliance reportmay provide information regarding, for example, the compliance postureof a computing resource when the computing resource is tested againstthe policy. However, prior compliance measurement systems generate thecompliance reports based on the polling (e.g., sampling) and subsequentassessment of computing resources at discrete moments in time separatedby significant time intervals. That is, prior compliance measurementsystems may perform compliance assessments by polling computingresources periodically (e.g., every thirty minutes) and, thereby,generate compliance reports that measure compliance periodically (e.g.,every thirty minutes). As a result, changes in the compliance state of acomputing resource that occur between compliance assessments (e.g.,between the periodic polling of computing resources) are not reported inthe compliance report.

Unlike prior systems, examples disclosed herein enable streamingcompliance measurements continuously. For example, rather than measuringcompliance periodically as a batch operation that polls multiplecomputing resources in a virtual computing environment at the same timeto determine configuration change events, examples disclosed here maymeasure on a per-event basis as configuration changes occur in a virtualcomputing environment, as discussed in greater detail above inconnection with the example compliance assessor 302 (FIG. 3). As aresult, when comparing compliance measurements over time, compliancemeasurements disclosed herein correspond to a trace of compliancemeasurements (e.g., a stream of continuous compliance measurements)rather than discrete (or impulse) values.

FIG. 9 is a block diagram of an example implementation of the examplecompliance measurer 304 of the example compliance monitor 218 of FIGS. 2and 3 to measure compliance. In the illustrated example of FIG. 9, thecompliance measurer 304 includes an example results monitor 902, anexample score calculator 904, an example aggregator 912, and an examplescore comparator 920. In the illustrated example, the score calculator904 includes an example surprisal calculator 906, an example informationgain calculator 908 and an example scorer 910. The aggregator 912 of theillustrated example includes an example group aggregator 914, an exampletemporal aggregator 916, and an example policy aggregator 918.

In the illustrated example of FIG. 9, the compliance measurer 304includes the results monitor 902 to monitor changes to assessmentresults. For example, the results monitor 902 may receive a notificationfrom the example compliance assessor 302 (FIGS. 3 and 5) when thecompliance assessor 302 updates an assessment result in the exampleresults database 312 (FIG. 3). For example, an assessment result maychange from satisfied to fails (or unsatisfied). In some examples, theresults monitor 902 may monitor incoming data to the results database312 to determine if the results database 312 is being updated with a newresult. When the results monitor 902 receives notification of a newresult (e.g., via the example compliance assessor 302), the resultsmonitor 902 determines the computing resource(s) affected by the resultschange and initiates the score calculator 904 to calculate a compliancescore for the computing resource in accordance with a policy (orpolicies).

In the illustrated example of FIG. 9, the compliance measurer 304includes the example score calculator 904 to calculate a compliancescore of a computing resource for a policy. For example, the scorecalculator 904 may retrieve a policy from the example policy database310 (FIG. 3) that is enforced against the computing resource. Thecompliance score (or measure of compliance) may be used to determine theamount of compliance, may be combined with other compliance scores(e.g., over time, over groups of computing resources and/or with otherpolicies), and/or may be used to compare two or more compliance scores.

The score calculator 904 of the illustrated example uses properties of apolicy to calculate a maximum surprisal for the policy and a remainingsurprisal value of the policy for a computing resource. The maximumsurprisal varies for each policy and depends on the properties of thepolicy. For example, the number of criteria and/or the logical structureof a policy may influence the maximum surprisal. Thus, the remainingsurprisal varies for each resource with respect to a policy and maydepend on the properties of the policy and/or the properties of theresource (e.g., the number of criteria, the logical structure of thepolicy, the truth values (e.g., true or false) resulting from testingthe resource against the policy criterion, etc.). The example scorecalculator 904 uses the difference between the maximum surprisal andremaining surprisal values (e.g., the information gain) as the measureof the amount of compliance to the policy by the computing resource. Inthe illustrated example, the score calculator 904 normalizes theinformation gain by dividing it by the maximum surprisal to calculate acompliance score of the policy for the computing resource. In addition,in some examples, the example score calculator 904 appends a timestampto the compliance score representative of when the compliance score wascalculated. In some examples, the score calculator 904 stores thecalculated compliance score in the example scores database 314 of FIG.3.

In the illustrated example of FIG. 9, the score calculator 904 includesthe example surprisal calculator 906 to calculate surprisal values. Forexample, the surprisal calculator 906 may use Equation 3a below tocalculate the maximum surprisal (I(P|0)) value of a policy (P). Thesurprisal value of satisfying a policy (P) is measured in criteria bits(sometimes referred to herein as “crits” or “bits”). A criteria bit is aunit of information when measuring a surprisal value in a binomialsystem (e.g., true/false, heads/tails, on/off, etc.)

$\begin{matrix}{{I\left( P \middle| 0 \right)} = {\log_{2}\frac{2^{\# {(P)}}}{{SatCount}(P)}}} & {{Equation}\mspace{14mu} 3a} \\{{I\left( P \middle| a \right)} = {\log_{2}\frac{2^{\# {({P|a})}}}{{SatCount}\left( P \middle| a \right)}}} & {{Equation}\mspace{14mu} 3b}\end{matrix}$

In Equation 3a above, the maximum surprisal (I(P|0)) is the surprisalvalue of the policy (P) given no (e.g., zero) amount of informationregarding the compliance status of any of the criteria of the policy.The SatCount(P) is the number of combinations of criteria truth-valuesthat satisfy the policy (P), and the value #(P) is the number ofdistinct checks in the policy (P). For example, referring back to thepolicy (P) defined in Equation 1 above and referring to FIG. 25A, thevalue #(P) is eight (e.g., (V₁, . . . V₈)) and the SatCount(P) is ninesince there are nine rows satisfying the policy (P) in FIG. 25A (i.e.,where the value of the policy (P) is true). There are 256 rows (i.e.,2^(#(P))=2⁸) in FIG. 25A, one for each combination of the truth-valuesof the eight checks (V₁ to V₈). Thus, using Equation 3a, the maximumsurprisal (I(P|0)) for the policy (P) defined in Equation 1 is equal to4.83 crits (i.e., log₂ 2⁸/9). To calculate the amount of remainingsurprisal (I(P|a)) for a policy (P) given a computing resource (a), thesurprisal calculator 906 calculates the SatCount(P|a) based on thecriteria (V₁, . . . V₈) satisfied by the computing resource (a) and usesEquation 3b above to calculate the surprisal of the computing resource(a). For example, referring back to the policy (P) defined in Equation 1above and referring to FIG. 25B, suppose a computing resource (a)satisfies criteria (¬V₁, V₅, and V₆), then the value #(P|a) is five(i.e., #(P)−3=8−3) and the SatCount(P|a) is the number of combinationsof truth values that satisfy the policy (P) chosen from amongst thosethat already satisfy the criteria (¬V₁, V₅, and V₆). Accordingly, theSatCount(P|a) is six, and the remaining surprisal (I(P|a)) for thepolicy (P) on the computing resource (a) is 2.42 crits (i.e., log₂2⁵/6).

In the illustrated example of FIG. 9, the score calculator 904 includesthe example information gain calculator 908 to calculate the informationgain (or the amount of compliance) of a policy for a computing resource.For example, the information gain calculator 908 may use Equation 4below to calculate the information gain.

G(P|a)=I(P|0)−I(P|a)  Equation 4:

In Equation 4 above, the information gain (G(P|a)) measures the amountof compliance that the computing resource (a) has with the policy (P)given the state of the computing resource (a). Thus, as the number ofsatisfied criteria in a policy grows, the information gain increases.Moreover, the information gain increases because as more of the criteriatruth-values are known (e.g., satisfied or unsatisfied by a computingresource), the surprisal value of the computing resource decreases(e.g., there is less uncertainty with whether the computing resource isable to satisfy the policy). In addition, in the illustrated example ofFIG. 9, if the compliance state for a computing resource changes (e.g.,switches from in-compliance to out-of-compliance) the cause for thecompliance change can be identified because a change in the informationgain may be attributed to a change in the truth-value of a criterion.Thus, credit or blame can be directly assigned to events (e.g.,configuration changes).

In the illustrated example of FIG. 9, the score calculator 904 includesthe example scorer 910 to calculate a compliance score of a policy for acomputing resource. For example, the scorer 910 may use Equation 5 belowto calculate a compliance score.

$\begin{matrix}{{B\left( P \middle| a \right)} = {\frac{G\left( P \middle| a \right)}{I\left( P \middle| 0 \right)}*100\%}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

02 In Equation 5 above, the compliance score (B(P|a)) is the amount thatthe computing resource (a) is in compliance with a policy (P) measuredin percentage. In Equation 5 above, the example scorer 910 divides theinformation gain (G(P|a)) by the maximum surprisal (I(P|0)) to normalizethe information gain. The example scorer 910 multiplies the result by“100%” to transform the range of values for the compliance score tobetween 0% and 100%.

In the illustrated example of FIG. 9, the compliance measurer 304includes the aggregator 912 to combine compliance scores. For example, asystem administrator may wish to know compliance of a policy enforcedagainst the example computing servers 104 (FIG. 1) of the examplevirtual computing environment 100 (FIG. 1). The example group aggregator914 uses Equation 6 below to calculate a combined compliance score(e.g., a group-aggregated compliance score for the retrieved compliancescores.

$\begin{matrix}{{B\left( P \middle| \left\{ {a_{1},{\ldots \mspace{14mu} a_{n}}} \right\} \right)} = \frac{\sum{G\left( P \middle| a_{j} \right)}}{n*{I\left( P \middle| 0 \right)}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

In Equation 6 above, the combined compliance score (B(P|{a₁, . . .a_(n)})) is the arithmetic average compliance score for the group ofcomputing resources ({a₁, . . . a_(n)}) against the policy (P). Forexample, the aggregator 912 and/or the group aggregator 914 may retrievethe compliance score for each of the computing resources in the groupfrom the example scores database 314. The combined compliance score forthe group of computing resources is then calculated by taking theaverage of the retrieved compliance scores (e.g., summing the compliancescores and dividing by the number of computing resources). As thecompliance scores for each of the computing resources is calculated withrespect to the same maximum surprisal value, the example groupaggregator 914 may take the arithmetic average of the compliance scoresfor the group of computing resources to calculate the combinedcompliance score (e.g., the group-aggregated compliance score) for thegroup of computing resources against the policy.

In the illustrated example of FIG. 9, the temporal aggregator 916 of theaggregator 912 combines compliance scores for a computing resource overtime. For example, a system administrator may want to know how acomputing resource measured in compliance over a period such as a weekor a historical range (e.g., Q1 of 2012). In some such examples, thetemporal aggregator 916 uses either Equation 7 below or Equation 8 belowto calculate a combined compliance score (e.g., a temporally-aggregatedcompliance score) for a computing resource over a group of discretemoments or over an interval, respectively.

$\begin{matrix}{{{B\left( P \middle| \left\{ {{a@t_{1}},{\ldots \mspace{14mu} {a@t_{m}}}} \right\} \right)} = \frac{\sum_{t}{G\left( P \middle| {a@t} \right)}}{m\;/\left( P \middle| 0 \right)}},{t\mspace{14mu} {in}\mspace{14mu} \left\{ {t_{1},{\ldots \mspace{14mu} t_{m}}} \right\}}} & {{Equation}\mspace{14mu} 7} \\{{{B\left( P \middle| {a@\left\lbrack {t_{1},t_{m}} \right\rbrack} \right)} = \frac{\int{{G\left( P \middle| {a@t} \right)}{dt}}}{\left( {t_{m} - t_{1}} \right)/\left( P \middle| 0 \right)}},{t\mspace{14mu} {in}\mspace{14mu} \left\{ {t_{1},t_{m}} \right\}}} & {{Equation}\mspace{14mu} 8}\end{matrix}$

In Equation 7 above, the combined compliance score (B(P{a@t₁, . . .a@t_(m)})) is calculated for a group of discrete moments ({t₁ . . .t_(m)}). In Equation 8 above, the combined compliance score(B(P|a@[t₁,t_(m)])) is calculated over an interval ([t₁,t_(m)]). In someexamples, the example aggregator 912 and/or the temporal aggregator 916may retrieve compliance scores for the computing resource (a) over aperiod from the example scores database 314. When the compliance scoresare of discrete moments, the example temporal aggregator 916 usesEquation 7 above to calculate the combined compliance score (e.g., thetemporally-aggregated compliance score) of the computing resource overtime. For example, the temporal aggregator 916 may retrieve compliancescores from the example scores database 314 corresponding to assessmentresults calculated during batch testing (e.g., on the first day of everymouth of the year). In some such examples, the assessment results arestored in the example results database 312 (FIG. 3) periodically (e.g.,every twenty-four hours), and, thus, the compliance scores retrievedfrom the scores database 314 correspond to discrete moments (e.g., everythirty days). In contrast, if the compliance scores are determinedcontinuously over time, then the example temporal aggregator 916 usesEquation 8 above to calculate the combined compliance score of thecomputing resource over time. For example, the assessment results storedin the results database 312 may correspond to event-driven assessments.In some such examples, assessment results are stored in the resultsdatabase 312 when an event is recorded. Thus, the correspondingcompliance scores retrieved from the scores database 314 are continuous(e.g., the compliance score at a time (t_(i)) is the same as at aprevious time (t_(i-1)) until an event is detected).

In the illustrated example of FIG. 9, the policy aggregator 918 of theaggregator 912 combines compliance scores for a computing resource overa collection of policies. For example, a collection of policies (P₁ toP_(n)) may be enforced on a computing resource (a). When calculating acombined compliance score for the computing resource (a) against thecollection of policies, the combined compliance score is not calculatedby arithmetically averaging the individual compliance scores ({B(P₁|a),. . . B(P_(n)|a)}) if the compliance scores are normalized by differentmaximum surprisals ({I(P₁|a), . . . I(P_(n)|a)}). For example, considera computing resource (a) that is 99% in-compliance against a firstpolicy (P₁) (i.e., compliance score (B(P₁|a))=99%) with a maximumsurprisal of two crits and that the computing resource (a) is 50%in-compliance against a second policy (P₂) (i.e., compliance score(B(P₂|a))=49%) with a maximum surprisal of 100 crits. Calculating thecombined compliance score using the arithmetic average would lead asystem administrator to believe that the computing resource (a) is 74%(i.e., (99+49)/2) in-compliance with the policies (P₁,P₂). However, theexample policy aggregator 918 calculates the combined compliance scoreby summing the independent gains for the policies (P₁,P₂) and thenrenormalizing the compliance score. Thus, as disclosed in greater detailbelow in connection with Equations 9-14, the example policy aggregator918 calculates a combined compliance score for the computing resource(a) of 50% (i.e., (0.99*2)/(100+2)+(0.49*100)/(100+2)).

As described above, policies may include a large number of criteria. Asa result, when combining compliance scores for a computing resource overa collection of policies, some policies may overlap (e.g., sharecriteria). In some such examples, the criteria dependencies may beaccounted for by calculating the combined compliance score as theconjunction of the collection of policies (e.g., the set of thecriteria). In the illustrated example, the policy aggregator 918 usesEquations 9-11 below to calculate a combined compliance score (e.g., apolicy-aggregated compliance score) for a computing resource against acollection of policies (P₁ to P_(n)) that include shared criteria.

$\begin{matrix}{\mspace{79mu} {{I\left( {P_{1}\bigcup\; {\ldots \mspace{14mu} P_{n}}} \middle| a \right)} = {I\left( {P_{1}\&\&{\ldots \mspace{14mu} P_{n}}} \middle| a \right)}}} & {{Equation}\mspace{14mu} 9} \\{{G\left( {P_{1}\bigcup\; {\ldots \mspace{14mu} P_{n}}} \middle| a \right)} = {{I\left( {P_{1}\bigcup\; {\ldots \mspace{14mu} P_{n}}} \middle| 0 \right)} - {I\left( {P_{1}\bigcup\; {\ldots \mspace{14mu} P_{n}}} \middle| a \right)}}} & {{Equation}\mspace{14mu} 10} \\{\mspace{79mu} {{B\left( {P_{1}\bigcup{\ldots \mspace{14mu} P_{n}}} \middle| a \right)} = \frac{G\left( {P_{1}\bigcup{\ldots \mspace{14mu} P_{n}}} \middle| a \right)}{I\left( {P_{1}\bigcup{\ldots \mspace{11mu} P_{n}}} \middle| 0 \right)}}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

In Equation 9 above, the combined surprisal value (I(P₁∪ . . . P_(n)|a))is calculated as the surprisal of the single policy (P₁&&P₂ . . . P_(n))formed from the conjunction (e.g., the logical AND operator) of thecollection of policies ({P₁, . . . P_(n)}). The example policyaggregator 918 uses Equation 10 above to calculate the information gain(G(P₁∪ . . . P_(n)|a)) for the computing resource (a) against thecollection of policies (P₁ to P_(n)). The example policy aggregator 918uses Equation 11 above to calculate the combined compliance score (B(P₁∪. . . P_(n)|a)) (e.g., the policy-aggregated compliance score) for thecomputing resource (a) against the collection of policies ({P₁, . . .P_(n)}). Thus, calculating a combined compliance score for a collectionof policies that share some criteria is similar to calculating acompliance score for a single policy formed from the collection ofpolicies ({P₁, . . . P_(n)}). As a result, the surprisal values, theinformation gain, and the compliance scores for the collection ofpolicies may be calculated using the same approach as described above inconnection with Equations 3-5, but with a larger policy formed from theconjunction of the policies of the collection of policies.

While Equations 9-11 above assume the collection of policies ({P₁, . . .P_(n)}) share (e.g., overlap with) some criteria, the example policyaggregator 918 may also calculate the combined compliance score for acollection of policies as if the policies are independent from eachother, even if the policies do share criteria. In some such examples,the policy aggregator 918 uses Equations 12-14 below to calculate acombined compliance score (e.g., a policy-aggregated compliance score)for a computing resource over a collection of policies.

$\begin{matrix}{\mspace{79mu} {{I\left( \left\{ {P_{1},\; {\ldots \mspace{14mu} P_{n}}} \right\} \middle| a \right)} = {\sum{I\left( P_{i} \middle| a \right)}}}} & {{Equation}\mspace{14mu} 12} \\{{G\left( \left\{ {P_{1},{\ldots \mspace{14mu} P_{n}}} \right\} \middle| a \right)} = {{I\left( \left\{ {P_{1},{\ldots \mspace{14mu} P_{n}}} \right\} \middle| 0 \right)} - {I\left( \left\{ {P_{1},\; {\ldots \mspace{14mu} P_{n}}} \right\} \middle| a \right)}}} & {{Equation}\mspace{14mu} 13} \\{\mspace{79mu} {{B\left( \left\{ {P_{1},{\ldots \mspace{14mu} P_{n}}} \right\} \middle| a \right)} = \frac{G\left( \left\{ {P_{1},{\ldots \mspace{14mu} P_{n}}} \right\} \middle| a \right)}{I\left( \left\{ {P_{1},{\ldots \mspace{11mu} P_{n}}} \right\} \middle| 0 \right)}}} & {{Equation}\mspace{14mu} 14}\end{matrix}$

When the collection of policies are independent of each other (orassumed to be independent of each other), the example policy aggregator918 calculates the combined surprisal value (I({P₁, . . . P_(n)}|a)) bysumming the surprisal values for each of the policies in the collectionof policies. As a result, the policy aggregator 918 uses Equation 12above to calculate the maximum surprisal (I(P_(i)|0)) for each of thepolicies (P_(i)) of the collection of policies ({P₁, . . . P_(n)}) andsums the individual maximum surprisals. The example policy aggregator918 also uses Equation 12 above to also calculate the remainingsurprisal value (I(P_(i)|a)) for the computing resource (a) for each ofthe policies (P_(i)) of the collection of policies ({P₁, . . . P_(n)})and sums the individual remaining surprisal values. Once the combinedsurprisal values are calculated, the example policy aggregator 918calculates the compliance score using the same approach as describedabove in connection with Equations 4 and 5. That is, the example policyaggregator 918 uses Equation 13 above to calculate the information gain(G({P₁, . . . P_(n)}a)) for the computing resource (a) against thecollection of policies ({P₁, . . . P_(n)}). The example policyaggregator 918 uses Equation 14 to calculate the combined score (B({P₁,. . . P_(n)}|a)) (e.g., the policy-aggregated compliance score) for thecomputing resource (a) against the collection of policies ({P₁, . . .P_(n)}).

In some examples, a combined compliance score for policies that overlapand a combined compliance score for policies that are independent may berelated by the mutual surprisal between the policies. For example,Equation 15 below may be used to relate dependent and independentcombined compliance scores of two policies (P_(x),P_(y)) against thecomputing resource (a).

$\begin{matrix}{{{I\left( {P_{x}\bigcup P_{y}} \middle| a \right)} = {{{I\left( \left\{ {P_{x},P_{y}} \right\} \middle| a \right)} + {I\left( {P_{x};\left. P_{y} \middle| a \right.} \right)}} = {{I\left( \left\{ {P_{x},P_{y}} \right\} \middle| a \right)} + 0}}}\mspace{79mu} \left( {{{if}\mspace{14mu} P_{x}},{P_{y}\mspace{14mu} {are}\mspace{14mu} {independent}}} \right)} & {{Equation}\mspace{14mu} 15}\end{matrix}$

In Equation 15 above, the combined surprisal value ((P_(x)∪P_(y)|a)) isthe conjunction of the two policies (P_(x),P_(y)) for the computingresource (a) and the surprisal (I(P_(x);P_(y)|a)) is the mutualsurprisal between the policies (P_(x),P_(y)). Thus, if the policies areindependent, then by using Equations 9 and 12 above, it holds that thecombined surprisal value I(P_(x)∪P_(y)|a)=I({P_(x),P_(y)}|a) as there isno mutual surprisal and the surprisal value (I(P_(x);P_(y)|a)) is equalto zero.

As discussed above, combining compliance scores depends on the maximumsurprisal for the combined policies. When policies are considered to beof equal priority (even with unequal maximum surprisals), the examplepolicy aggregator 918 may use Equation 16 below to calculate the averagecombined compliance score.

$\begin{matrix}{{{\mu \; {B\left( \left\{ {P_{1},{\ldots \mspace{14mu} P_{n}}} \right\} \middle| a \right)}} = {\frac{m}{n}*{\left( {\prod_{j \in J}{B\left( P_{j} \middle| a \right)}} \right)\hat{}\left( \frac{1}{m} \right)}}}{{{{where}\text{:}\mspace{14mu} J} = \left\{ {i:{{B\left( P_{i} \middle| a \right)} > 0}} \right\}},{m = {J}}}} & {{Equation}\mspace{14mu} 16}\end{matrix}$

In Equation 16 above, the average combined compliance score (μB({P₁, . .. P_(n)}|a)) is calculated as the expected (m of n) geometric mean ofthe (m) non-zero compliance scores in the set (J) with size (∥J∥). Thus,when using Equation 16 above to calculate an average combined compliancescore, the example policy aggregator 918 removes any bias that mayotherwise occur from the respective policies having different maximumsurprisals (e.g., an inherent bias). In some examples, a subjective biasmay be added to a combined compliance score by providing a priorityvalue to a policy. For example, the priority value may be a numericfactor (k_(i)) assigned to the policy (P_(i)) (e.g., written ask_(i)P_(i)). In some such examples, the policy aggregator 918 may useEquation 17 below to calculate an average combined compliance scorewhere the policies (P_(i)) are assigned a respective bias (k_(i)).

$\begin{matrix}{{{\mu \; {B\left( {\bigcup{k_{i}P_{i}}} \middle| a \right)}} = {\frac{m}{n}*{\left( {\prod_{j \in J}{{B\left( P_{j} \middle| a \right)}\hat{}k_{j}}} \right)\hat{}\left( \frac{1}{m} \right)}}}{{{{where}\text{:}\mspace{14mu} J} = \left\{ {i:{{B\left( P_{i} \middle| a \right)} > 0}} \right\}},{m = {\sum k_{j}}},{n = {\sum k_{i}}}}} & {{Equation}\mspace{14mu} 17}\end{matrix}$

In Equation 17 above, the bias (k_(i)) is the priority of the respectivepolicy (P_(i)) and the bias (k_(j)) is the priority of the respectivepolicy (P_(j)) having a non-zero score.

In the illustrated example of FIG. 9, the compliance measurer 304includes the example score comparator 920 to compare compliance scores.For example, a system administrator may wish to compare a computingresource (a_(j)) to a group of computing resources ({a₁, . . . a_(n)})to determine how the computing resource measures in compliance withrespect to policy (P) and the group of computing resources. In some suchexamples, the score comparator 920 retrieves the compliance score(B(P|a_(j))) for the computing resource (a_(j)) and the combinedcompliance score (B(P|{a₁, . . . a_(n)})) for the group of computingresources ({a₁, . . . a_(n)}) from the example scores database 314. Insome examples, the score comparator 920 may initiate the exampleaggregator 912 to calculate a combined score for the group of computingresources ({a₁, . . . a_(n)}). The example score comparator 920 comparesthe retrieved compliance scores (e.g., from the scores database 314and/or the aggregator 912) and stores the comparison results (e.g., avalue indicative of the difference between the compliance scores) in theexample scores database 314.

While the examples discussed above correspond to measuring compliance ofa policy by attempting to satisfy the criteria of the policy, theexample compliance measurer 304 may also calculate alternativemeasurements. For example, in some policies, the criteria may correspondto an undesired (e.g., pathological) state. That is, rather thanmeasuring compliance, the example compliance measurer 304 may measurenon-compliance of an inverse policy. For example, the example scorecalculator 904 may use Equation 18 below to convert a compliance scoreof a computing resource (a) on a policy (P) to the score of thecomputing resource (a) on the inverse policy (¬P).

B(P|a)=−B(¬P|a)  Equation 18:

In Equation 18 above, an improving compliance score occurs byprogressing towards 100% on the policy (P). Alternatively, an improvingcompliance score may also occur by progressing towards −100% on theinverse policy (¬P). That is, the compliance score improves as thecomputing resource (a) dissatisfies criteria of the inverse policy (¬P).For example, a policy may require that logging be disabled on acomputing resource. In some such examples, the compliance score for thecomputing resource improves when the “logging.enabled” state of thecomputing resource is “false.”

As discussed above in connection with surprisal values, the compliancescore is representative of the remaining amount of uncertainty (e.g.,missing information) that a computing resource can satisfy a policygiven the compliance state of the computing resource. In some examples.Equation 19 below may be used to define the set of truth-values ofcriteria (V_(a)) determined by a computing resource (a) on a policy (P).

V _(a) ={V→t:V∈V _(p)&& V(a)}  Equation 19:

The criteria truth-values that are determined using Equation 19 aboveare those that satisfy criteria: the set of criteria truth-values(V_(a)) contains values for the criteria (V) that are tested by a policy(P) (i.e., V∈V_(p)) and are satisfied by a computing resource (a) (i.e.,V(a)). Alternatively, when a policy corresponds to an inverse policy,Equation 20 below may be used to define the set of criteria truth-values(V_(a)) determined by a computing resource (a) on a policy (¬P).

V _(a) ={V→ƒ:V∈V _(p)&&¬V(a)}  Equation 20:

The criteria truth-values that are determined using Equation 20 aboveare those of unsatisfied criteria: the set of criteria truth-values(V_(a)) contains the criteria (V) that are tested by a policy (P) (i.e.,V∈V_(p)) and are unsatisfied by a computing resource (a) (i.e., ¬V(a)).

While Equation 19 and Equation 20 above may be used to calculate the setof determined criteria (V_(a)), the example score calculator 904 mayalso presume a value for the set (V_(a)) and use that set to calculatecompliance scores for hypothetical situations (e.g., “what if”compliance scores). For example, consider Equation 21 below, which is alogical representation of a policy (P).

P=(V ₁ ∥V ₂)&& V ₃  Equation 21:

In Equation 21 above, the number of criteria is three, the SatCount(P)is three (e.g., P(V₁,V₂,V₃) if (V₁→ƒ,V₂→t,V₃→t) or (V₁→t,V₂→ƒ,V₃→t) or(V₁→t,V₂→t,V₃→t)) and the maximum surprisal (I(P|0)) is 1.415 crits.Using Equations 5 and 21 above, the example score calculator 904 maycalculate the compliance score for different sets of criteria compliancestates. For example, Table 1 below illustrates the effect on thecompliance score when some criteria are considered established (presumedor determined). Each row illustrates an example of established criteria,the remaining undetermined criteria, the SatCount, and the resultingcompliance score, for the policy in Equation 21 above.

TABLE 1 Established Criteria Number of Undetermined Compliance V_(a)Criteria SatCount Score V₃ → t 2 3 70.7% V₃ → t, 1 1 29.3% V₂ → f V₃ →t, 0 0 −100%  V₂ → f, V₁ → f V₃ → t, 0 1  100% V₂ → f, V₁ → t

When the truth-values of a subset of the criteria (V_(p)) is establishedby a set (V_(a)), as discussed above in connection with the examplescore calculator 904, the maximum surprisal (I(P|0)) reflects the sizeof the policy in crits, while the information gain (G(P|V_(a))) measuresthe portion of the policy (P) satisfied by the established criteria(V_(a)), and the compliance score (1−B(P|V_(a))) is the normalizedamount of non-compliance with the policy (P). In addition, the impact ofa second set of established criteria truth-values (V′_(a)) to the policy(P) with respect to the first set of criteria truth-values (V_(a)) maybe represented by Equation 22 below. The percent difference between thesecond set of criteria truth-values (V′_(a)) and the first set ofcriteria (V_(a)) with respect to the policy (P) may be represented byEquation 23 below.

Impact=G(P|V′ _(a))−G(P|V _(a))  Equation 22:

Percent Difference=∥B(P|V′ _(a))−B(P|V _(a))∥  Equation 23:

In some examples, the dependence or independence of a policy may bedetermined based on a comparison of the maximum surprisals of thepolicies. For example, if Equation 24 below is true, then the twopolicies (P₁,P₂) are independent. If Equation 25 below is true, then thetwo policies (P₁,P₂) are dependent (e.g., overlap). If Equation 26 belowis true, then satisfying policy (P₁) is more uncertain (e.g., requiressatisfying more criteria). If Equation 27 below is true, then satisfyingthe first policy (P₁) subsumes (e.g., includes) the second policy (P₂).

I(P ₁&& P ₂|0)=I(P ₁|0)+I(P ₂|0)  Equation 24:

I(P ₁&& P ₂|0)<1(P ₁|0)+1(P ₂|0)  Equation 25:

I(P ₁|0)>(P ₂|0)  Equation 26:

I(P ₁|0)=I(P ₁|0)+I(P ₂|0)  Equation 27:

In some examples, Equations 24-27 above are also applicable to rules, asa rule may be considered a mini-policy.

While an example manner of implementing the compliance measurer 304 ofFIG. 3 is illustrated in FIG. 9, one or more of the elements, processesand/or devices illustrated in FIG. 9 may be combined, divided,re-arranged, omitted, eliminated and/or implemented in any other way.Further, the example results monitor 902, the example score calculator904, the example surprisal calculator 906, the example information gaincalculator 908, the example scorer 910, the example aggregator 912, theexample group aggregator 914, the example temporal aggregator 916, theexample policy aggregator 918, the example score comparator 920 and/or,more generally, the example compliance measurer 304 of FIG. 3 may beimplemented by hardware, software, firmware and/or any combination ofhardware, software and/or firmware. Thus, for example, any of theexample results monitor 902, the example score calculator 904, theexample surprisal calculator 906, the example information gaincalculator 908, the example scorer 910, the example aggregator 912, theexample group aggregator 914, the example temporal aggregator 916, theexample policy aggregator 918, the example score comparator 920 and/or,more generally, the example compliance measurer 304 could be implementedby one or more analog or digital circuit(s), logic circuits,programmable processor(s), application specific integrated circuit(s)(ASIC(s)), programmable logic device(s) (PLD(s)) and/or fieldprogrammable logic device(s) (FPLD(s)). When reading any of theapparatus or system claims of this patent to cover a purely softwareand/or firmware implementation, at least one of the results monitor 902,the example score calculator 904, the example surprisal calculator 906,the example information gain calculator 908, the example scorer 910, theexample aggregator 912, the example group aggregator 914, the exampletemporal aggregator 916, the example policy aggregator 918 and/or theexample score comparator 920 is/are hereby expressly defined to includea tangible computer readable storage device or storage disk such as amemory, a digital versatile disk (DVD), a compact disk (CD), a Blu-raydisk, etc. storing the software and/or firmware. Further still, theexample compliance measurer 304 of FIG. 3 may include one or moreelements, processes and/or devices in addition to, or instead of thoseillustrated in FIG. 9, and/or may include more than one of any or all ofthe illustrated elements, processes and devices.

Flowcharts representative of example machine-readable instructions forimplementing the compliance measurer 304 of FIGS. 3 and/or 9 are shownin FIGS. 10-14 and 26. In this example, the machine-readableinstructions comprise a program for execution by a processor such as theprocessor 2412 shown in the example processor platform 2400 discussedbelow in connection with FIG. 24. The programs may be embodied insoftware stored on a tangible computer readable storage medium such as aCD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), aBlu-ray disk, or a memory associated with the processor 2412, but theentire program and/or parts thereof could alternatively be executed by adevice other than the processor 2412 and/or embodied in firmware ordedicated hardware. Further, although the example programs are describedwith reference to the flowcharts illustrated in FIGS. 10-14 and 26, manyother methods of implementing the example compliance measurer 304 mayalternatively be used. For example, the order of execution of the blocksmay be changed, and/or some of the blocks described may be changed,eliminated, or combined. Although the operations in the flow diagrams ofFIGS. 10-14 and 26 are shown in seriatim, the examples of FIGS. 10-14and 26 may additionally or alternatively be implemented so that one ormore of the operations are performed in parallel with one or more otherones of the operations.

The example program of FIG. 10 begins at block 1002 when the examplecompliance measurer 304 (FIGS. 3 and 9) receives a notification for achange in an assessment result. For example, the results monitor 902(FIG. 9) may receive a notification from the example compliance assessor302 (FIG. 3) or the example results database 312 (FIG. 3) of a change inan assessment result. In some examples, the change in an assessmentresult may correspond to a configuration change in a computing resource(e.g., the example computing server 104 (FIG. 1)) in the example virtualcomputing environment 100 (FIG. 1). At block 1004, the example scorecalculator 904 (FIG. 9) calculates a compliance score. In some examples,block 1004 is implemented using an example method described below inconnection with FIG. 11.

At block 1006, the compliance measurer 304 determines whether to combinethe calculated compliance score with one or more compliance scores. Forexample, the compliance score calculated in block 1004 may be includedwith compliance scores for a group of computing resources that are alsosubject to compliance policies. If, at block 1006, the examplecompliance measurer 304 determines to combine the compliance scorecalculated at block 1004 with one or more compliance scores, then, atblock 1008, the example aggregator 912 (FIG. 9) combines compliancescores over a group of assets, over time, or over a collection ofpolicies. Example processes that may be used to implement block 1008 isdescribed below in connection with FIGS. 12-14. Control then proceeds toblock 1010 to determine whether to compare compliance scores.

After the aggregator 912 combines compliance scores over a group ofassets, over time, or over a collection of policies at block 1008, or ifthe compliance measurer 304 determines not to combine the calculatedcompliance score at block 1006, at block 1010, the compliance measurer304 determines whether to compare the calculated compliance score withone or more compliance scores. For example, the compliance scorecalculated in block 1004 may be compared to other scores calculated inblock 1004 and/or combined at block 1008.

If, at block 1010, the compliance measurer 304 determines to compare thecalculated compliance score, then, at block 1012, the example scorecomparator 920 compares compliance scores. For example, the scorecomparator 920 may calculate the difference between a first computingresource compliance score and a second computing resource compliancescore. In some examples, the score comparator 920 may determine whetherone compliance score is within a threshold of a second compliance score.For example, the score comparator 920 may determine whether the firstcomputing resource is operating within a threshold (e.g., apredetermined threshold such as 5%, a dynamic threshold that changesbased on the time of day, etc.) of a combined compliance score of thecomputing resource over time (e.g., determine whether the firstcomputing resource is operating within a threshold of its usualcompliance scores). In some examples, the score comparator 920 mayidentify a trend of compliance of the first computing resource over timebased on compliance scores determined at respective times. In someexamples, the score comparator 920 stores the results of the comparisonin the scores database 314. For example, the score comparison resultsmay be used to identify moments in time when a computing resource is notin compliance with a policy. Control then proceeds to block 1014 todetermine whether to continue measuring compliance.

After the score comparator 920 compares compliance scores at block 1012,or if the compliance measurer 304 determines not to compare compliancescores at block 1010, then, at block 1014, the compliance measurer 304determines whether to continue measuring compliance. For example, acompliance assessment process may be stopped, or power to the compliancemeasurer 304 may be removed. In some such examples, the example programof FIG. 10 ends. Otherwise, if the compliance measurer 304 determines tocontinue measuring compliance, control returns to block 1002, and theexample compliance measurer 304 continues to measure compliance of thevirtual computing environment 100.

The example method of FIG. 26 determines compliance of a computingresource when a notification for a change in an assessment result is notreceived. For example, the example method of FIG. 26 may be executedwhen the example compliance measurer 304 (FIGS. 3 and 9) is measuringcompliance but the results monitor 902 (FIG. 9) does not receive anotification from the example compliance assessor 302 (FIG. 3) or theexample results database 312 (FIG. 3) of a change in an assessmentresult. The example method of FIG. 26 may be used to implement block 404of FIG. 4. The example method of FIG. 26 begins at block 2602 when theexample score comparator 920 (FIG. 9) retrieves compliance scores tocompare. For example, the score comparator 920 may retrieve one or morecompliance scores calculated at block 1004 (FIG. 10), combined at block1008 (FIG. 10) and/or compared at block 1012 (FIG. 10) from the exampleresults database 312.

At block 2604, the compliance measurer 304 determines whether to combinecompliance scores. For example, one of the retrieved compliance scoresmay be included in a group of computing resources that are also subjectto compliance policies. If, at block 2604, the example compliancemeasurer 304 determines to combine at least one of the retrievedcompliance scores at block 2602 with one or more other compliancescores, then, at block 2606, the example aggregator 912 (FIG. 9)combines the compliance scores over a group of assets, overtime, or overa collection of policies. Example processes that may be used toimplement block 2606 is described below in connection with FIGS. 12-14.Control then proceeds to block 2608 to determine whether to comparecompliance scores.

After the aggregator 912 combines compliance scores over a group ofassets, over time, or over a collection of policies at block 2606, or ifthe compliance measurer 304 determines not to combine compliance scoresat block 2604, at block 2608, the compliance measurer 304 determineswhether to compare compliance score(s) with other compliance score(s).For example, the combined compliance score calculated in block 2606 maybe compared to other compliance scores retrieved in block 2602.Alternatively, if the example compliance measurer 304 determines not tocombine compliance scores at block 2604, for example, the compliancemeasurer 304 may determine whether to compare a first compliance scoreretrieved in block 2602 to other compliance scores retrieved in block2602.

If, at block 2608, the compliance measurer 304 determines to comparecompliance scores, then, at block 2610, the example score comparator 920compares compliance scores. For example, the score comparator 920 maycalculate the difference between a first computing resource compliancescore and a second computing resource compliance score. In someexamples, the score comparator 920 may determine whether one compliancescore is within a threshold of a second compliance score. For example,the score comparator 920 may determine whether the first computingresource is operating within a threshold (e.g., a predeterminedthreshold such as 5%, a dynamic threshold that changes based on the timeof day, etc.) of a combined compliance score of the computing resourceover time (e.g., determine whether the first computing resource isoperating within a threshold of its usual compliance scores). In someexamples, the score comparator 920 may identify a trend of compliance ofthe first computing resource over time based on compliance scoresdetermined at respective times. In some examples, the score comparator920 stores the results of the comparison in the scores database 314. Forexample, the scored comparison results may be used to identify momentsin time when a computing resource is not in compliance with a policy.Control then proceeds to block 2612 to determine whether to continuemeasuring compliance.

After the score comparator 920 compares compliance scores at block 2610,or if the compliance measurer 304 determines not to compare compliancescores at block 2608, then, at block 2612, the compliance measurer 304determines whether to continue measuring compliance. For example, acompliance assessment process may be stopped, or power to the compliancemeasurer 304 may be removed. In some such examples, the example programof FIG. 26 ends. Otherwise, if the compliance measurer 304 determines tocontinue measuring compliance, control returns to block 2602, and theexample compliance measurer 304 continues to measure compliance of thevirtual computing environment 100.

The example method of FIG. 11 calculates a compliance score for acomputing resource. For example, a system administrator may decide tocalculate a compliance score for a computing resource (a_(j)) (e.g., theexample computing server 104 of FIG. 1) of a set of computing resources({a₁, . . . a_(m)}) against a policy (P_(i)) of a collection of policies({P₁, . . . P_(n)}). The example method of FIG. 11 may be used toimplement block 1004 of FIG. 10. The example method of FIG. 11 begins atblock 1102 when the example surprisal calculator 906 (FIG. 9) of thescore calculator 904 (FIG. 9) determines a maximum surprisal value((I(P_(i)|0)) and a remaining surprisal value (I(P_(i)|a_(j))) for acomputing resource (a_(j)). For example, the surprisal calculator 906may use Equation 3a above to calculate a maximum surprisal ((I(P_(i)|0))for the policy (P_(i)). The example surprisal calculator 906 may alsouse Equation 3b above to calculate a remaining surprisal value(I(P_(i)|a_(j))) for the computing resource (a_(j)) against the policy(P_(i)).

At block 1104, the example information gain calculator 908 (FIG. 9)determines the information gain based on the calculated surprisalvalues. For example, the information gain calculator 908 may useEquation 4 above to measure the amount of compliance (G(P_(i)|a_(j)))with the policy (P_(i)) given the configuration state of the computingresource (a_(j)).

At block 1106, the example scorer 910 (FIG. 9) determines a compliancescore for the computing resource. For example, the scorer 910 may useEquation 5 above to calculate a compliance score (B(P_(i)|a_(j))) forthe computing resource (a_(j)) against the policy (P_(i)).

At block 1108, the example score calculator 904 determines whether todetermine another compliance score. For example, the score calculator904 may determine that the notification received from the resultsmonitor 902 includes another computing resource (a_(j)) or may determinethat another policy (P_(i)) is enforced against the computing resource(a_(j)). If the score calculator 904 determines that there is anothercompliance score to calculate, control returns to block 1102 tocalculate new surprisal values. Otherwise, if, at block 1108, the scorecalculator 904 determines not to calculate another compliance score,control returns to a calling function or process such as the exampleprogram of FIG. 10, and the example process of FIG. 11 ends.

The example method of FIG. 12 combines compliance scores for a group ofcomputing resources against a policy. The example method of FIG. 12 maybe used to implement block 1010 of FIG. 10. The example method of FIG.12 begins at block 1202 when the example aggregator 912 (FIG. 9)retrieves compliance scores from the example scores database 314 (FIG.3). For example, the aggregator 912 may retrieve compliance scores forESXi hosts, for virtual machines, for ESX hosts that are part of acluster, for physical computing devices like networking and storagedevices, etc.

At block 1204, the example group aggregator 914 (FIG. 9) sums theinformation gain (G(P_(i)|{a₁, . . . a_(n)})) of the compliance scoresfor each of the computing resources included in the group of computingresources ({a₁, . . . a_(n)}). In some examples, the group aggregator914 may cause the information gain calculator 908 (FIG. 9) to calculatean information gain for each of the computing resources in the group ofcomputing resources.

At block 1206, the example group aggregator 914 determines the combinedcompliance score for the group of computing resources. For example, thegroup aggregator 914 may use Equation 6 above to normalize the summedinformation gain (ΣG(P_(i)|a_(j))) using the maximum surprisal value(I(P_(i)|0)) for the policy to calculate the combined compliance score(B(P_(i)|{a₁, . . . a_(n)})) for the group of computing resources ({a₁,. . . a_(n)}). In some examples, the group aggregator 914 may triggerthe example surprisal calculator 906 (FIG. 9) to calculate the maximumsurprisal ((P_(i)|0)) for the policy (P_(i)).

At block 1208, the example group aggregator 914 stores the calculatedcombined compliance score (e.g., the group-aggregated compliance score)in the example scores database 312. Control then returns to a callingfunction or process such as the example program of FIG. 10, and theexample process of FIG. 12 ends.

The example method of FIG. 13 combines compliance scores for a computingresource over time. The example method of FIG. 13 may be used toimplement block 1010 of FIG. 10. The example method of FIG. 13 begins atblock 1302 when the example aggregator 912 (FIG. 9) retrieves compliancescores from the example scores database 314 (FIG. 3). For example, theaggregator 912 may retrieve compliance scores for a computing resource(a_(j)) during a past period.

At block 1304, the example temporal aggregator 916 (FIG. 9) determineswhether the retrieved compliance scores are of discrete moments orresulting from configuration change events continuously collected inreal-time as configuration changes occur. For example, the aggregator912 may retrieve compliance scores over the course of a week (e.g., aduration during which continuous monitoring of events is used to detectevents as configuration changes occur), every Monday at 9:00 AM (e.g.,batch polling at discrete moments in time), etc. If the temporalaggregator 916 determines that the retrieved compliance scorescorrespond to configuration change events of discrete moments at times({t₁, . . . t_(m)}), at block 1306, the temporal aggregator 916 sums theinformation gain (G(P|a@t) for j in (1 . . . m) of the compliance scorescorresponding to a time during which a batch collection process wasperformed. In some examples, the temporal aggregator 916 causes theinformation gain calculator 908 to calculate an information gain for thecomputing resource (a) for a particular time (t_(j)) (e.g.,corresponding to each time at which a batch collection process wasperformed).

At block 1308, the example temporal aggregator 916 determines thecombined compliance score for the computing resource based onconfiguration change events. For example, the temporal aggregator 916may use Equation 7 above to normalize the summed information gain(ΣG(P|a@t_(i))) using the maximum surprisal (I(P|0)) for the policy (P)to calculate the combined compliance score (B(P|{a@t₁, . . . a@t_(m)}))for the discrete time moments ({t₁, . . . t_(m)}). In some examples, thetemporal aggregator 916 may cause the example surprisal calculator 906to calculate the maximum surprisal for the policy.

At block 1310, the example temporal aggregator 916 stores the calculatedcombined compliance score (e.g., the temporally-aggregated compliancescore) of the computing resource (a) over the period ({t₁ . . . t_(m)})in the example scores database 314.

Referring again to block 1304, if the example temporal aggregator 916determines that the retrieved compliance scores correspond toconfiguration change events collected continuously over time (e.g., overa continuous interval of time [t₁,t_(m)]), then, at block 1312, thetemporal aggregator 916 integrates the information gain (G(P|a@t)) ofthe retrieved compliance scores for the computing resource (a) over theinterval [t₁,t_(m)]. For example, the temporal aggregator 916 may causethe information gain calculator 908 to calculate an information gain forthe computing resource over the duration of continuously collectedconfiguration change events. In some examples, the temporal aggregatorcauses the information gain calculator 908 to calculate the informationgain at points of configuration change (e.g., upon change events). Thetemporal aggregator 916 may then use Equation 8 above to normalize theintegrated information gain to calculate the combined compliance score.Control then proceeds to block 1308 to calculate a combined compliancescore. For example, the temporal aggregator 916 may use Equation 8 aboveto normalize the integrated information gain to calculate the combinedcompliance score (B(P|a@[t₁,t_(m)])). At block 1310, the exampletemporal aggregator 916 stores the calculated combined compliance scoreof the computing resource (a) over the period [t₁,t_(m)] in the examplescores database 314. Control then returns to a calling function orprocess such as the example program of FIG. 10, and the example processof FIG. 13 ends.

The example method of FIG. 14 combines compliance scores for a computingresource over a collection of policies. The example method of FIG. 14may be used to implement block 1010 of FIG. 10. The example method ofFIG. 14 begins at block 1402 when the example aggregator 912 (FIG. 9)retrieves compliance scores from the example scores database 314 (FIG.3). For example, the aggregator 912 may retrieve compliance scores for acomputing resource (a) calculated with respect to policies from acollection of policies ({P₁, . . . P_(n)}). At block 1404, the examplepolicy aggregator 918 (FIG. 9) determines whether the retrievedcompliance scores are calculated against policies that overlap. Forexample, the policies overlap if they have at least one rule in common(e.g., two policies measuring the same category such as risk, compliancehealth, etc.).

If the policy aggregator 918 determines the policies associated with theretrieved compliance scores do overlap (e.g., share one or morecriteria), then the combined compliance score is calculated as theconjunction of the policies and, at block 1406, the policy aggregator918 determines the surprisal values for the conjunction of the policies.That is, the combined compliance score for the collection of policiesmay be calculated similar to calculating the compliance score for apolicy including all the criteria of the collection of policies. Forexample, the policy aggregator 918 may use Equation 9 above to calculatethe maximum surprisal (I(P₁∪ . . . P_(n)|0)) of the collection ofpolicies ({P₁, . . . P_(n)}). The example policy aggregator 918 may useEquation 9 above to also calculate the remaining surprisal value (I(P₁∪. . . P_(n)|a)) of the computing resource (a) against the collection ofpolicies ({P₁, . . . P_(n)}). In some examples, the policy aggregator918 may cause the surprisal calculator 906 (FIG. 9) to calculate thesurprisal values.

At block 1408, the example policy aggregator 918 determines theinformation gain for the conjunction of the collection of policies. Forexample, the policy aggregator 918 may use Equation 10 (FIG. 9) tocalculate the information gain (G(P₁∪ . . . P_(n)|a)) for the collectionof policies ({P₁, . . . P_(n)}) enforced against the computing resource(a_(i)). In some examples, the policy aggregator 918 may cause theinformation gain calculator 908 (FIG. 9) to calculate the informationgain.

At block 1410, the example policy aggregator 918 determines the combinedcompliance score for the computing resource (a) over the collection ofpolicies ({P₁, . . . P_(n)}). For example, the policy aggregator 918 mayuse Equation 11 above to normalize the information gain (G(P₁∪ . . .P_(n)|a)) using the maximum surprisal (I(P₁∪ . . . P_(n)|0)) for theconjunction of the collection of policies ({P₁, . . . P_(n)}) tocalculate the combined compliance score (B(P₁∪ . . . P_(n)|a)) for thecomputing resource (a) against the collection of policies ({P₁, . . .P_(n)}). In some examples, the policy aggregator 918 may cause theexample surprisal calculator 906 (FIG. 9) to calculate the maximumsurprisal (I(P₁∪ . . . P_(n)|0)) for the conjunction of the policies({P₁, . . . P_(n)}).

At block 1412, the example policy aggregator 918 stores the calculatedcombined compliance score (e.g., the policy-aggregated compliance score)in the example scores database 314.

Returning to block 1404, if the example policy aggregator 918 determinesthe policies associated with the retrieved compliance scores do notoverlap (e.g., are independent policies sharing no criteria, or are tobe treated as independent policies, etc.), then, at block 1414, thepolicy aggregator 918 determines the combined surprisal value as the sumof the surprisal values of all the policies in the collection ofpolicies ({P₁, . . . P_(n)}). For example, the policy aggregator 918 mayuse Equation 12 above to calculate the maximum surprisal (I(P|0)) foreach policy in the collection of policies ({P₁, . . . P_(n)}). Theexample policy aggregator 918 may use Equation 12 above to alsocalculate the surprisal value (I({P₁, . . . P_(n)}|a)) for the computingresource against each of the policies in the collection of policies({P₁, . . . P_(n)}). In some examples, the policy aggregator 918 maycause the surprisal calculator 906 to calculate the surprisal values.

At block 1416, the example policy aggregator 918 determines theinformation gain for the computing resource against the collection ofpolicies. For example, the policy aggregator 918 may use Equation 13above to calculate the combined information gain (G({P₁, . . .P_(n)}|a)). In some examples, the policy aggregator 918 may cause theinformation gain calculator 908 to calculate the information gain.

At block 1418, the example policy aggregator 918 determines the combinedcompliance score for the computing resource over the collection ofpolicies. For example, the policy aggregator 918 may use Equation 14above to normalize the combined information gain (G({P₁, . . . P_(n)}a))using the summed maximum surprisal value (Σ(I(P_(i)|0))) for thecollection of policies ({P₁, . . . P_(n)}) against the computingresource (a). In some examples, the policy aggregator 918 may cause theexample surprisal calculator 906 to calculate the summed maximumsurprisal value for the collection of policies. Control then proceeds toblock 1412, and the example policy aggregator 918 stores the calculatedcombined compliance score in the example scores database 314. Afterblock 1412, control returns to a calling function or process such as theexample program of FIG. 10, and the example process of FIG. 14 ends.

III. Identifying Priorities

Compliance policies apply to an inventory of computing resources such asthe computing resources of the example virtual computing environment 100of FIG. 1. Policies may be defined by any organization, industryconsortium, government agency, company and/or any other entity orindividual. Some example policies that may be used with examplesdisclosed herein are provided by VMware, Inc., a Delaware corporation,in the vSphere Hardening Guide (VHG) and by the Defense InformationSecurities Agency (DISA) in the form of Secure Technical ImplementationGuides (STIGS).

An individual rule may be applicable to a subset of the inventory, suchas to ESXi hosts within a datacenter that contains ESXi hosts,datastores and networking equipment. Thus, as used herein, an assetclass is the subset of an inventory of computing resources satisfying ascope condition (e.g., satisfying a test performed by the scope tester406 of FIG. 4) of a rule within a policy. The set of computing resources(or computing assets) that satisfy the scope condition may sometimes bereferred to as being within the scope of the rule. For example, a firstasset class may include all ESXi hosts, a second asset class may includeVirtual Machines (VMs) with VMware Tools software installed, and a thirdasset class may include ESX hosts that are members of a cluster. In someexamples, asset classes may overlap (e.g., a resource may be a member ofmore than one asset class). The size of an asset class (e.g., the firstasset class, the second asset class, etc.) is the number of computingresources it contains. For example, the VHG policy identifies 27 assetclasses tested by 150 rules.

To determine compliance of a computing resource that is within the scopeof a rule, the computing resource is tested against a rule checkcondition (e.g., a test performed by the check tester 408 of FIG. 4) ofthe rule. In some examples, when a scope condition fails (e.g., thecomputing resource is not within the scope of the rule), the rulecorresponding to the scope condition is considered inapplicable, a rulecheck condition corresponding to the rule is not tested, and no defectis generated. When the computing resource satisfies the scope conditionbut fails the rule check condition of the rule, a defect is generated.Thus, as used herein, a defect results (e.g., is generated) when acomputing resource fails the rule check condition corresponding to arule. When a defect is generated, the associated computing resource andrule, including the scope condition results and the rule checkcondition, may be logged. In the illustrated example, each defect has anassociated repair action to correct the defect. For example, a repairaction may include installing an update or a software patch. In someexamples, an update or patch changes a configuration setting such assetting an “enable copying” property of a computing resource to False tosatisfy a “disable copy” rule of a policy.

Using prior systems to test large numbers of computing resources using asignificant number of rules, the resulting list of defects (sometimesreferred to as a “compliance report” or “results”) may quickly becomeunmanageable. Fully automated remediation using such prior systems isoften impractical. Examples disclosed herein are useful to overcome suchlimitations of prior systems by ranking defects so that high-prioritydefects may be prioritized or ranked relatively higher to receivequicker attention and/or remediation. Thus, to process defects resultingfrom computing resources that fail policy rules, example methods andapparatus disclosed herein prioritize defects according to theirrelevance to particular users (e.g., system administrators), entities(e.g., organizations or businesses), etc.

Relevance, as used herein, may be represented as a priority orderdetermined by past (i.e., historical) repair actions performed by thecorresponding person, entity, etc. For example, response times of howquickly users addressed and/or repaired the same types of defects may berepresented as defect historical repair rates. A defect historicalrepair rate indicates how quickly defects have been addressed by theuser (e.g., an entity, an IT department, an individual, etc.) in thepast. For example, a user may historically address (e.g., attempt torepair) defects associated with risk categories (e.g., securityvulnerabilities) quicker than defects associated with operationalcompliance categories. Examples disclosed herein combine the defecthistorical repair rate with weights learned from the defect repairhistory (e.g., a log of past repair actions to repair defects) to mapthe defect historical repair rate into a priority order that correlateswith relevance. To this end, example methods and apparatus disclosedherein include grouping defects into defect classes that share a commonrepair action to correct the defect, generating an estimation functionbased on previous user behavior and attentiveness at maintaining apolicy, estimating a priority order for the defect classes using theestimation function, and ranking the defect classes based on theirestimated priority order.

FIG. 15 is a block diagram of an example implementation of the exampleresults prioritizer 306 (FIG. 3) of the example compliance monitor 218of FIGS. 2 and 3. In the illustrated example of FIG. 15, the exampleresults prioritizer 306 includes an example results grouper 1502, anexample repairs database 1504, an example defect class database 1510, anexample repair bias calculator 1520, an example repair effectivenesscalculator 1522 and an example priority estimator 1524. The examplerepairs database 1504 includes an example repairs map 1506 that maps adefect to a repair action used to correct the defect. The examplerepairs database 1504 also includes an example past repair actionsdatabase 1508 that stores a repair history (e.g., historical data) ofpast repair actions. For example, the past repair actions database 1508may identify when a defect was detected and when a repair action tocorrect the defect was executed (or implemented) by a user (e.g., asystem administrator).

In the illustrated example of FIG. 15, the example results prioritizer306 includes the example results grouper 1502 to aggregate defects intogroupings (e.g., defect classes). When the example results grouper 1502receives a notification of a defect (e.g., from the example complianceassessor 302 (FIG. 3), the example results database 312 (FIG. 3), etc.),the results grouper 1502 uses the scope test information included in thedefect to identify an asset class that includes the computing resourceidentified in the defect. In the illustrated example of FIG. 15, theresults grouper 1502 queries the repairs map 1506 for a repair action tocorrect the defect.

The example results grouper 1502 of FIG. 15 aggregates defects intodefect classes by grouping the defects sharing a common asset class andhaving the same repair action. Defects in the same defect classcorrespond to computing resources of the same asset class. Thus, therepair action (e.g., installing a patch) associated with those defectsis applicable to and may be used for any computing resource in the assetclass. For example, the results grouper 1502 may group a first defectinto a defect class based on the first defect belonging to a same assetclass and corresponding to a same repair action as a second defect inthe defect class (e.g., two different defects fixed by applying the samepatch). In some examples, instead of prioritizing individual resource(e.g., asset) defects, the example results grouper 1502 enablesprioritizing defect classes, which facilitates using the same repair toprocess (e.g., correct) the defects in the same defect class. The sizeof a defect class is the number of defects contained in the defectclass. The extent of a defect class is representative of the set ofcomputing resources affected by the defects in the defect class althougha one defect to one resource mapping may not be true for each instance(e.g., a resource may have two defects in the same class). In examplesdisclosed herein, defects are associated with a timestamp to identifydifferent times at which defects occur. After the example resultsgrouper 1502 classifies a defect, the results grouper 1502 logs a defectentry corresponding to the classified defect in the defect classdatabase 1510.

In the illustrated example of FIG. 15, the example defect class database1510 stores defect entries generated by the example results grouper 1502when classifying an assessment result indicative of a compliance failure(e.g., a defect). An example entry 1512, shown in FIG. 15, includes anexample defect identifier 1513, an example timestamp 1514, an exampleasset identifier 1515, an example asset class identifier 1516, anexample policy rule identifier 1517, an example repair action identifier1518, and an example defect class identifier 1519. The example defectidentifier 1513 identifies a defect (e.g., a defect reference number)that resulted from an assessment. The example timestamp 1514 identifiesthe time at which the defect identified by the defect identifier 1513occurred. The example asset identifier 1515 identifies a computingresource that corresponds to the defect identified by the defectidentifier 1513. The example asset class identifier 1516 identifies anasset class that includes the computing resource identified by the assetidentifier 1515. The example policy rule identifier 1517 identifies therule failed by the computing resource identified by the asset identifier1515. The example repair action identifier 1518 identifies a repairaction to correct the defect identified by the defect identifier 1513.The example defect class identifier 1519 identifies a defect class thatincludes the defect identified by the defect identifier 1513. Theexample Table 2 below includes three defect entries (e.g., defects)categorized into two example defect classes having corresponding policyrules, asset classes and repair actions based on the VHG policydiscussed above.

TABLE 2 Defect Class Rule Asset Class Repair 1 disable-console- VM withisolation.tools.copy.disable := copy Tools True 2 isolate-vum- vCenterLimit the connectivity between proxy Update Manager and public patchrepositories. 2 isolate-vum- vCenter Limit the connectivity betweenwebserver Update Manager and public patch repositories.

In the illustrated example of FIG. 15, the example results prioritizer306 includes the example repair bias calculator 1520 to determine arepair bias for preferring to repair defects of one defect class overdefects of another defect class. For example, the example repair biascalculator 1520 may determine that a first type of defect class maycontain defects having shorter detect-to-repair times (e.g., theduration between defect detection and defect repair and/or defect repairinitiation) than other defect classes. In some such examples, the repairbias calculator 1520 may determine that the user exhibits quickerresponse times for correcting defects of the first defect class than forcorrecting defects from other defect classes.

The example repair bias calculator 1520 of FIG. 15 uses the historicalrate of repair (e.g., retrieved from the example past repair actionsdatabase 1508) for each of the defect classes based on historical repairactions to determine prioritizations for correcting different defectsfrom different defect classes. That is, the example results prioritizer306 may utilize the defect class historical rate of repairs calculatedby the example repair bias calculator 1520 to identify defectsassociated with a historical higher rate of repair (e.g., shorterintervals between defect detection and defect repair) as defects that auser responds to more quickly relative to other defects from defectclasses with lower rates of repair.

In the illustrated example of FIG. 15, the example repair biascalculator 1520 determines the historical rate of repair for a defectclass based on previous assessment results from the example resultsdatabase 310 of FIG. 3, the example defect class database 1510 and/orthe example past repair actions database 1508. For example, the repairbias calculator 1520 may identify when a defect was detected and when arepair action to correct the defect was executed (or implemented) by auser from the example past repair actions database 1508. Thus, the rateof repair may be representative of an average historicaldetect-to-repair time between detecting a same defect as a first defectand initiating repair of the same defect. The techniques disclosed belowin connection with Equations 28 and 29 may be used by the example repairbias calculator 1520 to calculate a repair bias for repairing defectsfrom one defect class over defects from another defect class. Forexample, the repair bias calculator 1520 may use Equation 28 below tocalculate the rate (y) at which a defect (in a defect class (D_(j))) isrepaired at a time (t).

y _(j) ^(t)=min((∥A _(j) ^(t−1) ∥−∥A _(j) ^(t)∥)/∥C _(j)∥,0)  Equation28:

In Equation 28 above, an asset class (C_(j)) is a subset of an inventoryof computing resources (e.g., assets) that satisfy a scope condition fora policy rule, and the extent (A_(j)) of a defect class (D_(j)) isrepresentative of the set of computing resources affected by the defectsin a defect class (D_(j)). The fraction of an asset class (C_(j))affected by a defect class (D_(j)) is the ratio of the number ofcomputing resources (∥A_(j)∥) affected by the defect class (D_(j)) tothe number of computing resources in the asset class (∥C_(j)∥). Thus,the example repair bias calculator 1520 uses Equation 28 above tocalculate the change in the fraction of an asset class affected bydefects in a defect class (D_(j)) between assessment tests (e.g.,between times (t−1,t)). In some examples, the repair bias calculator1520 may then use Equation 29 below to calculate the normalized rate(y_(j)) at which defects in a defect class (D_(j)) are repaired.

y _(j)=Σ_(t) y _(j) ^(t) /n, for n time cycles  Equation 29:

In Equation 29 above, the number of time cycles (n) is the differenttimes data points were calculated by Equation 28 above. Thus, theexample repair bias calculator 1520 uses Equation 29 above to calculatethe average rate of repair for a defect in a defect class (D_(j)) (e.g.,the normalized rate (e.g., detect-to-repair time per defect occurrence)at which defects in a defect class (D_(j)) are repaired) per step (e.g.,between assessment tests to generate the defects). In the illustratedexample, the average rate of repair (y_(j)) is representative of therepair bias of defect class (D_(j)). In some examples, the average rateof repair (y_(j)) may be used to determine relevance. For example, adefect class with a relatively higher average rate of repair (y_(j))(e.g., a lower detect-to-repair time per defect occurrence) maycorrespond to a more relevant defect to repair to a user, and, thus, maybe prioritized or ranked relatively higher than a defect class with alower average rate of repair.

In the illustrated example of FIG. 15, the example results prioritizer306 includes the example repair effectiveness calculator 1522 tocalculate the effect that repairing a defect has on the overallcompliance of computing resources with a policy. For example, correctinga defect by changing a property state (or configuration setting) of acomputing resource affects all policies that include that computingresource in their scope. This change is quantifiable, and the examplerepair effectiveness calculator 1522 of FIG. 15 calculates an averagechange in compliance score (e.g., a repair effectiveness) for eachpolicy resulting from repairs to a defect class. The repaireffectiveness may then be used to prioritize certain repairs overothers. For example, a defect class with a relatively higher repaireffectiveness may correspond to a more relevant defect to repair to auser, and, thus, may be prioritized or ranked relatively higher than adefect class with a lower repair effectiveness.

In the illustrated example of FIG. 15, the example repair effectivenesscalculator 1522 may retrieve compliance scores from the examplecompliance measurer 304 (FIG. 3) and/or the example scores database 314(FIG. 3) that are indicative of a degree of compliance of a computingresource (e.g., an asset) to a policy. For example, for each policy (P)and computing resource (a), a compliance score of B(P|a) in the range[−1,1] is calculated, where 1 indicates total satisfaction of the policyby the computing resource (a). As discussed in greater detail above inconnection with the example compliance measurer 304, the compliancescore may be normalized for groups of computing resources and/or of acomputing resource at specific times. For example, the compliance score(B(P|{a₁, . . . a_(j)})) is the compliance score of a policy (P) giventhe computing resources ({a₁, . . . a_(n)}). The compliance score ofcomputing resources may be utilized to determine the score of computingresource (a) on policy (P) at time (t) (e.g., B(P|a^(t))). In some suchexamples, an average change in compliance score (e.g., scoreimprovement) when repairs to defects in a defect class (D_(j) ^(t)) areperformed may be found using Equations 30 and 31 below.

$\begin{matrix}{b_{j,i}^{t} = {\min \left( {{{B\left( P_{i} \middle| A_{j}^{t} \right)} - {B\left( P_{i} \middle| A_{j}^{t - 1} \right)}},0} \right)}} & {{Equation}\mspace{14mu} 30} \\{{b_{j,i} = \frac{\sum_{t}b_{j,i}^{t}}{n}},{{for}\mspace{14mu} n\mspace{14mu} {cycles}}} & {{Equation}\mspace{14mu} 31}\end{matrix}$

In Equation 30 above, the change in compliance score (b_(j,i) ^(t)) isthe difference in compliance score between the extent of the defectclass at a first time (t) and at a previous time (t−1), the compliancescore (B(P_(i)|A_(j) ^(t))) is the compliance score for the policy(P_(i)) given the extent of the defect class at the first time (t), andthe compliance score (B(P_(i)|A_(j) ^(t−1))) is the compliance score forthe policy (P) given the extent of the defect class at the second time(t−1). In other words, the change in compliance score (b_(j,i) ^(t)) isthe amount that repairing a defect in a defect class (D_(j) ^(t))improves the compliance score on the policy (P_(i)) at time (t). Theexample repair effectiveness calculator 1522 may then use Equation 31above to calculate the repair effectiveness (b_(j,i)) on a policy(P_(i)) (e.g., the average (or normalized) change in compliance score(e.g., score improvement) on a policy (P_(i))) based on the average ofthe summed total of the change in compliance scores given a change inthe extent (A_(j)) (e.g., the set of affected computing resources) of adefect class per step (e.g., between assessment tests).

In the illustrated example of FIG. 15, the example results prioritizer306 includes the example priority estimator 1524 to estimate a rankorder for defect classes based on relevance to corrective actions thatare typically performed by a particular user (e.g., an entity, an ITgroup, an individual, and/or any other type of user of content). Ratherthan retrieving explicit preferences from a user (e.g., prompting theuser to identify a priority order), the example priority estimator 1524uses repair bias to estimate a rank order for the defect classes. Thatis, historical repair data such as past repair actions and previousassessment results are utilized to determine which, if any, defect classdefects a user implicitly prefers to correct relatively more quicklythan defects from other defect classes. In some such examples, thedefect classes may be ranked based on the average duration of a repairinterval. For example, a defect class with a high rate of repair (e.g.,a short detect-to-repair interval per defect occurrence) may berepresentative of types of defects that a user prefers to correctrelatively more quickly than other defects. Thus, the example priorityestimator 1524 may estimate a priority order of defect classes based onrepair bias calculations retrieved from the example repair biascalculator 1520. As a result, the past corrective behavior of the userin question is used to set the priority during as opposed to expresspriority rankings input by the user. In some examples, express priorityrankings may additionally or alternatively be entered. In some examples,the priority estimator 1524 facilitates displaying the defect class inrank order with a plurality of other defect classes based on theestimated priority.

As discussed above, repairing a defect may affect compliance scores ofmore than one policy that is enforced in the computing environment.Thus, in some examples, the priority estimator 1524 may estimate a rankorder for defect classes based on repair effectiveness of a repair inmaking a computing resource comply with (e.g., improve the compliancescore of) a policy. For example, the priority estimator 1524 may comparethe average score change step on a policy for the different defectclasses and rank-order the defect classes accordingly. Thus, the examplepriority estimator 1524 may estimate a priority order of defect classesbased on repair effectiveness calculations retrieved from the examplerepair effectiveness calculator 1522.

In some examples, the example priority estimator 1524 leverages the rateof repair calculations with the repair effectiveness calculations toestimate a priority order for the defect classes. In some such examples,the example priority estimator 1524 uses a learning algorithm (e.g., amachine-learning algorithm) to generate an estimation function based onhistorical data to estimate the priority order, which may then be usedto predict the rank-order for the defect classes.

The priority estimator 1524 of the illustrated example uses a supportvector regression (SVR) learning algorithm to generate an estimationfunction. Instead of soliciting training examples from a user, theexample priority estimator 1524 builds a training set based onhistorical data retrieved from, for example, the results database 310 ofFIG. 3, the example defect class database 1510 and/or the example pastuser repairs database 1518. As a result, the example SVR learningalgorithm learns to predict relevant priorities based on past repairactions and previous assessment results. In some such examples, thepriority estimator 1524 may use Equations 32 and 33 below to build atraining set for the example SVR learning algorithm.

S _(j) ^(t)=(x _(j) ^(t−1) ,y _(j) ^(t))  Equation 32:

S ^(t)=∪_(j) S _(j) ^(t)  Equation 33:

In Equation 32 above, the training set vector (S_(j) ^(t)) for defectsin a defect class (D_(j)) at time (t) is calculated using the rate ofrepair (y_(j) ^(t)) for defects in the defect class (D_(j)) at time (t)and a feature vector (x_(j) ^(t−1)). The feature vector (x_(j) ^(t−1))is a vector that combines the rate of repair calculations and the repaireffectiveness calculations for the time (t−1) into a vector. Forexample, the feature vector (x_(j) ^(t−1)) may include the averagerepair rate (y_(j) ^(t−1)) of defects in a defect class (D_(j)), therepair effectiveness (b_(j,i) ^(t−1)) on policies P_(i) in ({P₁ . . .P_(n)}) the asset class (C_(j)) corresponding to the defect class(D_(j)), etc. The example priority estimator 1524 uses the training setvector (S_(j) ^(t)) to build a training set (S^(t)), which may then beinput into the example SVR learning algorithm. As shown in Equation 33above, the training set (S^(t)) is the union of training set vectors(S_(j) ^(t)) for the defects in the defect class (D_(j)) at a time (t).Thus, the training set (S^(t)) represents a collection of featurevectors (x_(j) ^(t−1), y_(j) ^(t)) for each (j) and defect class (D_(j))at time (t).

The example priority estimator 1524 may then use the training set(S^(t)) as input into the SVR learning algorithm. For example, thepriority estimator 1524 may use Equation 34 below to generate a rankestimating function (ƒ^(t)).

ƒt=SVR(S ^(t))  Equation 34:

In Equation 34 above, the rank estimating function (ƒ^(t)) is output bythe SVR learning algorithm at each time (t) and the rank estimatingfunction (ƒ^(t)) depends on the training set input (S^(t)) to the SVRlearning algorithm. The example priority estimator 1524 of theillustrated example uses the rank estimating function (ƒ^(t)) to predictthe next (e.g., at time t+1) rate of repair (y_(j) ^(t+1)) based on thefeature vector (x_(j) ^(t)) for defects in the defect class (D_(j)). Forexample, the priority estimator 1524 may use Equation 35 below topredict the next rate of repair (y_(j) ^(t+1)).

y _(j) ^(t+1)≅ƒ^(t)(x _(j) ^(t))  Equation 35:

In Equation 35 above, the predicted next rate of repair (y_(j) ^(t+1))is representative of an estimated relevance based on past repairactions. In some examples, the predicted next rate of repair is used asa priority for repairing defects of defect class (D_(j)) at time (t)(e.g., D_(j) ^(t)). In some such examples, the priority estimator 1524may facilitate displaying the defect classes in rank order based on thepriorities for repairing the defects of the respective defect classes.

In the illustrated example of FIG. 15, the priority estimator 1524estimates relevance (e.g., priorities) of a defect based on the repairhistory of the corresponding defect class. Additionally, the calculatedrepair bias used by the priority estimator 1524 is user-agnostic. Thatis, a first user repair bias for repairing, for example, performancecompliance defects may influence the repair priorities predicted for asecond user (e.g., a user more interested in, for example, repairingsecurity vulnerability defects).

Accordingly, in some examples, the example priority estimator 1524 mayuse user identifiers to personalize the estimated relevance (y_(j)^(t+1)) and prediction function (ƒ^(t)) calculated in Equation 35 above.For example, when an entry is logged in the past repair actions database1508, the entry may also include a user identifier to identify the userwho initiated the repair. In some such examples, the user identifier maybe used to personalize the repair bias for repairing defects (e.g., tospecific IT personnel, specific groups in an entity, etc.). For example,the priority estimator 1524 may filter entries from the example pastrepair actions database 1508 that do not match the user identifierrepresentative of the current user prior to estimating a relevance(e.g., an estimated next rate of repair) for a defect class such thatonly repairs by the corresponding user affect the estimate. In some suchexamples, the repair bias of the first user would not skew the predictedrepair priorities for a second user.

In some examples, the priority estimator 1524 adjusts prioritizationbased on preferences of system administrators or groups within anenterprise that may work on different portion of the virtual computingenvironment 100 (FIG. 1) or on the same portion at different times(e.g., day shift versus night shift). For example, defect repairs madeby one member of a work-group may influence the expected rate of repair(and hence defect class relevance) of other members of the work-group.In some examples, the repair history may be weighted based on hierarchyin a group. For example, the priority estimator 1524 may give preferenceto (e.g., weigh heavier) a repair history of a system administrationdepartment manager than a system administrator in the department whenestimating relevance.

In some examples, defect classes may influence other defect classes. Forexample, different policy rules may share criteria (e.g., testedconfiguration settings). As a result, a first user preference forrepairing a first defect class may be used to influence the estimatednext rate of repair (y_(j) ^(t+1)) of a related second defect class thatshares criteria.

In some examples, a “filter bubble” may influence the estimatedrelevance (e.g., the estimated next rate of repair) of defect classes. Afilter bubble may occur when recommendations are influenced (e.g.,overwhelmingly influenced) by past behavior and, thereby, biasing theuser (or repairs) into more of the same behavior. For example, therepair history for a first defect class may be so robust that repairingsubsequent defects of the first defect class are continuouslyprioritized over repairing other defect classes. As a result, the otherdefect classes are unable to rise in priority, and the example priorityestimator 1524 continues to prioritize the first defect class over theother defect classes, thereby continuing to bias the user into repairingdefects in the first defect class. In some such examples, the inputsinto the priority estimator 1524 may be periodically (e.g., once perweek) and/or aperiodically (e.g., in response to a user request)modified based on user preferences. For example, a user may select(e.g., via a switch) to disable smart recommendations, rank defectsconventionally (e.g., not based on estimated relevance, etc.), insertrandom results (e.g., into the training data), and/or adjust the degreesof predictive power versus static scoring via a graphical user interfaceelement such as, for example, a “volume knob.”

While an example manner of implementing the results prioritizer 306 ofFIG. 3 is illustrated in FIG. 15, one or more of the elements, processesand/or devices illustrated in FIG. 15 may be combined, divided,re-arranged, omitted, eliminated and/or implemented in any other way.Further, the example results grouper 1502, the example repairs database1504, the example repairs map 1506, the example past repair actionsdatabase 1508, the example defect class database 1510, the examplerepair bias calculator 1520, the example repair effectiveness calculator1522, the example priority estimator 1524, and/or, more generally, theexample results prioritizer 306 of FIG. 3 may be implemented byhardware, software, firmware and/or any combination of hardware,software and/or firmware. Thus, for example, any of the example resultsgrouper 1502, the example repairs database 1504, the example repairs map1506, the example past repair actions database 1508, the example defectclass database 1510, the example repair bias calculator 1520, theexample repair effectiveness calculator 1522, the example priorityestimator 1524 and/or, more generally, the example results prioritizer306 could be implemented by one or more analog or digital circuit(s),logic circuits, programmable processor(s), application specificintegrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s))and/or field programmable logic device(s) (FPLD(s)). When reading any ofthe apparatus or system claims of this patent to cover a purely softwareand/or firmware implementation, at least one of the example resultsgrouper 1502, the example repairs database 1504, the example repairs map1506, the example past repair actions database 1508, the example defectclass database 1510, the example repair bias calculator 1520, theexample repair effectiveness calculator 1522 and/or the example priorityestimator 1524 is/are hereby expressly defined to include a tangiblecomputer readable storage device or storage disk such as a memory, adigital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc.storing the software and/or firmware. Further still, the example resultsprioritizer 306 of FIGS. 3 and 15 may include one or more elements,processes and/or devices in addition to, or instead of, thoseillustrated in FIG. 15, and/or may include more than one of any or allof the illustrated elements, processes and devices.

Flowcharts representative of example machine-readable instructions forimplementing the results prioritizer 306 of FIGS. 3 and 15 are shown inFIGS. 16 and 17. In this example, the machine-readable instructionscomprise a program for execution by a processor such as the processor2412 shown in the example processor platform 2400 discussed below inconnection with FIG. 24. The programs may be embodied in software storedon a tangible computer readable storage medium such as a CD-ROM, afloppy disk, a hard drive, a digital versatile disk (DVD), a Blu-raydisk, or a memory associated with the processor 2412, but the entireprogram and/or parts thereof could alternatively be executed by a deviceother than the processor 2412 and/or embodied in firmware or dedicatedhardware. Further, although the example program is described withreference to the flowcharts illustrated in FIGS. 16 and/or 17, manyother methods of implementing the example results prioritizer 306 mayalternatively be used. For example, the order of execution of the blocksmay be changed, and/or some of the blocks described may be changed,eliminated, or combined. Although the operations in the flow diagrams ofFIGS. 16 and 17 are shown in seriatim, the examples of FIGS. 16 and 17may additionally or alternatively be implemented so that one or more ofthe operations are performed in parallel with one or more other ones ofthe operations.

The example method of FIG. 16 begins at block 1602 when the exampleresults grouper 1502 (FIG. 15) groups defects into defect classes. Inthe illustrated examples, the results grouper 1502 groups the defectsinto defect classes based on a same asset class and a same repairaction. For example, the results grouper 1502 may determine an assetclass for a defect based on a non-compliant computing resource, and theresults grouper 1502 may use the example repairs map 1506 (FIG. 15) todetermine a repair action for the defect. In some such examples, thedefect classes may be determined by the different combinations of assetclasses and repair actions associated with the defects.

At block 1604, the example priority estimator 1524 (FIG. 15) generatesan estimation function of relevance (e.g., the next rate of repair) forthe defect classes. In the illustrated example, the priority estimator1524 determines the estimation function of relevance based on historicaldata including previous assessment results and/or past repair actions.An example method that may be used to implement block 1604 is describedbelow in connection with FIG. 17. For example, the priority estimator1524 may use Equation 34, as discussed above in connection with FIG. 15,to generate the estimation function.

At block 1606, the priority estimator 1524 estimates a priority orderfor the defect classes using the estimation function. For example, thepriority estimator 1524 may estimate a priority order for each of thedefect classes listed in a backlog of unrepaired defects. The priorityestimator 1524 may then use Equation 35, as discussed above inconnection with FIG. 15, to estimate a priority order for the defectclasses with unrepaired defects and then determine a priority order forthe defect classes based on their estimated priority order.

At block 1608, the example results prioritizer 306 (FIGS. 3 and 15)determines whether a repair action is received. For example, the resultsprioritizer 306 may pause prioritizing defect classes until an action istaken against the ordered defect class. For example, the resultsprioritizer 306 may check whether a response to the estimated priorityorder is received from a user. If no repair action is received at block1608, then, at block 1610, the results prioritizer 306 determineswhether a timeout occurs. If no timeout occurs at block 1610, thencontrol returns to block 1608 to determine whether a repair action isreceived. Otherwise, if a timeout does occur at block 1610, then controlproceeds to block 1614 to determine whether to continue prioritizingdefect classes.

Returning back to block 1608, if the results prioritizer 306 determinesthat a repair action was received, then, at block 1612, the resultsprioritizer 306 updates the repair rates of defect classes. For example,the results prioritizer 306 may update the repair rates of the defectclasses stored in the past repair actions database 1508 (FIG. 15) basedon the repair action(s) taken at block 1608. Control then proceeds toblock 1614 to determine whether to continue prioritizing defect classes.

At block 1614, the example results prioritizer 306 determines whether tocontinue prioritizing defect classes. For example, a process orapplication using the results prioritizer 306 may be stopped or powermay be removed from the results prioritizer 306 (e.g., through a systemshutdown event). If the results prioritizer 306 determines to stopprioritizing defect classes, the example process of FIG. 16 ends.Otherwise, control returns to block 1602 to group defects into defectclasses.

The example method of FIG. 17 may be used to generate an estimationfunction of relevance based on historical data. The example method ofFIG. 17 may be used to implement block 1604 of FIG. 16. Initially, atblock 1702, the example repair bias calculator 1520 (FIG. 15) determinesa repair rate for repairing defects in a defect class based on previousassessment results and/or past repair actions. For example, the repairbias calculator 1520 may use Equation 28, as discussed above inconnection with FIG. 15, to calculate a change in the fraction ofcomputing resources affected by defects in a defect class (D_(j)) at atime (t). The example repair bias calculator 1520 may then use Equation29, as discussed above in connection with FIG. 15, to calculate anaverage rate of repair (y_(j)) at which defects in a defect class (D)are repaired based on the sum of the change in the number of computingresources affected by the defect class (D_(j)) over the number of cycles(n).

At block 1704, the example repair effectiveness calculator 1522 (FIG.15) determines repair effectiveness of repairing a defect for a policyenforced in a computing environment. For example, the repaireffectiveness calculator 1522 may use Equation 30, as discussed above inconnection with FIG. 15, to calculate a change in compliance score giventhe extent of the defect class at a first time and at a second time. Theexample repair effectiveness calculator 1522 may then use Equation 31,as discussed above in connection with FIG. 15, to calculate a repaireffectiveness (b_(j,i)) (e.g., an average change in compliance score)for each policy enforced in the computing environment resulting fromrepairs to defects in a defect class (D_(j)).

At block 1706, the example priority estimator 1524 generates anestimation function for unrepaired defects. An example technique ofgenerating the estimation function includes training a learningalgorithm (e.g., machine-learning algorithm) based on repair rates(e.g., the repair rates determined at block 1702) and repaireffectiveness (e.g., the repair effectiveness determined at block 1704).For example, the priority estimator 1524 may use Equations 32 and 33, asdiscussed above in connection with FIG. 15, to build a training setincluding the repair rates from the example repair bias calculator 1520and the repair effectiveness from the repair effectiveness calculator1522. In some examples, the priority estimator 1524 may then useEquation 34, as discussed above in connection with FIG. 15, to generatean estimation function for the unrepaired defects based on the output ofthe training algorithm (e.g., a support vector regression (SVR)algorithm).

At block 1708, the example priority estimator 1524 determines whether togenerate another estimation function. For example, the priorityestimator 1524 may receive notification that the example past repairactions database 1508 (FIG. 15) was updated with additional past repairactions. If the priority estimator 1524 determines there is anotherestimation function to generate, for example, based on the additionalpast repair actions, control returns to block 1702. Otherwise, if, atblock 1708, the priority estimator 1524 determines not to generateanother estimation function, control then returns to a calling functionor process such as the example program of FIG. 16, and the exampleprocess of FIG. 17 ends.

IV. Reporting Results

The example reporter 318 of FIG. 3 may retrieve user information storedin the example compliance database 308 (FIG. 3) to generate reports fora user (e.g., a system administrator). Reports may be helpful indetermining the compliance posture of a computing resource and/or thevirtual computing environment 100 (FIG. 1). The example reporter 318 ofFIG. 3 generates different reports based on different user selections.For example, the reporter 318 may retrieve assessment results stored inthe results database 312 (FIG. 3) and generate a report identifying theassessment results for one or more of computing resource(s) (e.g.,assets), a policy (or policies), the virtual computing environment 100,etc. In some examples, the reporter 318 retrieves compliance scores fromthe scores database 314 (FIG. 3) and generates a report identifying thecompliance scores for one or more of a computing resource(s), a policy(or policies), a range of time, a set of designated times, the virtualcomputing environment 100, etc. In some other examples, the reporter 318retrieves rankings from the priority order database 316 (FIG. 3) andgenerates a report identifying the ranked order of the assessmentresults.

FIGS. 18-23 illustrate example interfaces generated by the examplereporter 318. FIG. 18 illustrates an example interface 1800 displaying areal-time compliance summary for a computing environment (e.g., theexample virtual computing environment 100). The example interface 1800displays a compliance score 1802, a finance group identifier 1804, apolicy identifier 1806, and a compliance timeline 1808. The examplereporter 318 may retrieve the compliance score 1802 from the examplescores database 314. The example reporter 318 may retrieve the groupidentifier 1804 from the example core services controller 216 (FIG. 2)(e.g., via a resources and virtual machine inventory managementapplication). In the illustrated example, the group identifier 1804corresponds to a group of computing resources associated with a financedepartment. The example reporter 318 may retrieve a policy enforced inthe finance group of computing resources from the example policydatabase 310 (FIG. 3) and display the policy identifier 1806corresponding to the retrieved policy. The example reporter 318 mayretrieve previous compliance scores for the finance group of computingresources to generate the compliance timeline 1808.

FIG. 19 illustrates an example interface 1900 displaying a compliancesummary of a computing resource's compliance with a policy. The examplereporter 318 may retrieve an inventory list 1902 from the examplecompliance database 308 (FIG. 3) to display to a user. In theillustrated example, a computing resource or resource group 1904(CompRsc.eng.vmware.com) from the inventory list 1902 is selected, andthe policy 1906 (e.g., the vSpher 5 Hardening Guide) is enforced on thecomputing resource 1904. In the illustrated example, the populationsummary 1908 corresponds to the group of computing resources included inthe selected computing resource 1904. Thus, the reporter 318 mayretrieve assessment results from the example results database 312 (FIG.3) that are related to or under control of the selected computingresource 1904. In the illustrated example of FIG. 19, the populationsummary 1908 provides the number of compliance successes 1910 (e.g.,four), the number of compliance critical-failures 1912 (e.g., 669), andthe number of compliance warnings 1914 (e.g., three). In some examples,a compliance critical-failure 1912 may indicate that a computingresource is out-of-compliance with a critical rule. For example, acompliance critical-failure may be indicative of a securityvulnerability with the selected computing resource. In some examples, acompliance warning 1914 may be indicative of a computing resource thatis out-of-compliance with a non-critical rule (or non-essential rule).

In some examples, the compliance status (e.g., the compliance success,the compliance critical-failure and/or the compliance warning) maycorrespond to compliance score threshold percentages for the computingresource. For example, a compliance score between 90% and 100% maycorrespond to a compliance success, a compliance score between 70% and89% may correspond to a compliance warning, and a compliance scorebetween 0% and 69% (or less than 70%) may correspond to a compliancecritical-failure. Thus, in some examples, the number of compliancesuccesses 1910, the number of compliance critical-failures 1912, and thenumber of compliance warnings 1914 correspond to the number of computingresources within the group of computing resources included in theselected computing resource 1904 in which the compliance score waswithin one of the compliance score threshold percentages. In some otherexamples, the number of compliance successes 1910, the number ofcompliance critical failures 1912, and the number of compliance warnings1914 may correspond to the compliance score of one computing resourceassessed at different times ({t₁, . . . t_(m)}). In some other examples,the number of compliance successes 1910, the number of compliancecritical failures 1912, and the number of compliance warnings 1914 maycorrespond to the compliance score of one computing resource assessedover a period of time [t₁,t_(m)]. In some other examples, the number ofcompliance successes 1910, the number of compliance critical-failures1912, and the number of compliance warnings 1914 may correspond to thecompliance score of the different rules that are imposed on the selectedcomputing resource 1904. In some examples, the threshold percentages maybe modified by a user.

FIG. 20 illustrates an example interface 2000 displaying a compliancesummary for computing resources related to a selected computingresource. The example interface 2000 includes the inventory list 2002and identifies the selected computing resource 2004(CompRsc.eng.vmware.com). In the illustrated example, the reporter 318(FIG. 3) retrieves assessment results from the example results database312 (FIG. 3) and/or compliance scores from the example scores database314 (FIG. 3) that are related to the selected computing resource 2004.The example interface 2000 displays the compliance scores for a group ofvirtual machines 2006 included in the selected computing resource 2004.An example list of virtual machines 2006 and the correspondingcompliance scores is displayed in the interface panel 2008.

FIG. 21 illustrates an example interface 2100 displaying a combinedpopulation compliance summary for a group of computing resources relatedto a selected computing resource and a compliance summary for theselected computing resource. The example interface 2100 of FIG. 21includes an inventory list 2102, the selected computing resource 2104(CompRsc.eng.vmware.com), and the policy 2106 (e.g., the vSphere 5Hardening Guide) that is enforced on the computing resource 2104. Theexample interface 2100 also includes the population summary 2108corresponding to the group of computing resources included in theselected computing resource 2104 and the corresponding number ofcompliance successes 2110 (e.g., two), number of compliancecritical-failures 2112 (e.g., 58), and number of compliance warnings2114 (e.g., one). The example interface 2100 also includes interfacepanel 2116 to display a compliance summary for the selected computingresource 2104. In the illustrated example of FIG. 21, the interfacepanel 2116 includes indicators for compliance successes 2118 and forcompliance failures 2120 for corresponding ones of the rules 2122. Theexample interface panel 2116 also includes a severity indicator 2124 toindicate the level of severity (e.g., a compliance critical-failure 1912(FIG. 19), and/or a compliance warning 1914 (FIG. 19), etc.) when theselected computing resource 2104 is out-of-compliance with one of therules 2122.

FIG. 22 illustrates an example interface 2200 displaying detailedcompliance information for a selected computing resource. The exampleinterface 2200 displays an inventory list 2202, the selected computingresource 2204 (BobH2k8), and a “Details” tab 2206 that is selected. Theexample interface 2200 also includes a detailed compliance timeline2208. To generate the detailed compliance timeline 2208, the reporter318 (FIG. 3) may retrieve compliance scores from the example scoresdatabase 314 (FIG. 3). As discussed in greater detail above inconnection with the example compliance assessor 302 and the examplecompliance measurer 304 (FIG. 3), the compliance assessor 302 performscompliance assessments when an event (e.g., a configuration change) isdetected (e.g., the compliance assessor 302 is event-driven), and thecompliance measurer 304 calculates compliance scores when an assessmentresult changes. Thus, the compliance scores may be displayed in thedetailed compliance timeline 2208 as a trace 2210 (e.g., a stream ofcontinuous compliance measurements). In the illustrated example, thetrace 2210 is representative of the compliance score of the selectedcomputing resource 2204 overtime. In addition, marks 2212 located on thetrace 2210 are indicative of when compliance measurements wereperformed.

The example interface 2200 also includes example interface panel 2214 todisplay historical compliance measurements in a list. Accordingly, theexample interface panel 2214 includes compliance state (or status)indicators 2216 and severity indicators 2218 for the corresponding rules2220. The example interface panel 2214 also includes the policy 2222(sometimes referred to as “standard”) to which each of the rules 2220correspond and a timestamp 2224 indicative of when the compliancemeasurement was made. The example interface panel 2214 includes a changeindicator 2226 to indicate when a change in compliance for a rule 2220is measured. In the illustrated example, the rule 2230 is selected and apopup panel 2228 displays detailed information regarding the complianceassessment. In the illustrated example, the reporter 318 retrievesdetailed information regarding the rule from the example policy database310 (FIG. 3) (e.g., the “standard” (e.g., the policy), the “rule,” the“severity” level, the check “test” and the “expected value” (e.g., thecheck state). The example reporter 318 may retrieve the “found” state(e.g., the computing resource state) from the example results database312 (FIG. 3).

FIG. 23 illustrates an example interface 2300 to display assessmentresults for a policy. For example, the reporter 318 (FIG. 3) mayretrieve assessment results from the example results database 312 (FIG.3). In the illustrated example, the interface 2300 includes a detailspanel 2302 providing assessment results for compliance of the computingresource 2304 (e.g., the Bronze Datacenter) with the policy 2306 (e.g.,the VM Hardening Guide 5.1). The example details panel 2302 includes atimeline 2308 and a results list 2310. In the illustrated example, theexample timeline 2308 projects compliance measurements 2312 over athirty day period. For example, the reporter 318 may process compliancescores retrieved from the example scores database 314 (FIG. 3) and/orthe example score comparator 920 (FIG. 9) to identify trends incompliance measurements. The example reporter 318 may then predictcompliance measurements 2312 over a time period of interest. In theillustrated example of FIG. 23, the results list 2310 is a filtered listof assessment results. For example, the results list 2310 only includesassessment results indicative of compliance failures 2314 for rules witha critical severity level 2316. Thus, the example results list 2310 maydisplay a combination of assessment results corresponding to acompliance state 2314 that is selected (e.g., compliance failures,compliance successes, compliance unknown, and/or complianceinapplicable) and a rule severity level (e.g., critical, non-critical,etc.).

As mentioned above, the example processes of FIGS. 4, 6, 7, 10-14, 16and 17 may be implemented using coded instructions (e.g., computerand/or machine-readable instructions) stored on a tangible computerreadable storage medium such as a hard disk drive, a flash memory, aread-only memory (ROM), a compact disk (CD), a digital versatile disk(DVD), a cache, a random-access memory (RAM) and/or any other storagedevice or storage disk in which information is stored for any duration(e.g., for extended time periods, permanently, for brief instances, fortemporarily buffering, and/or for caching of the information). As usedherein, the term tangible computer readable storage medium is expresslydefined to include any type of computer readable storage device and/orstorage disk and to exclude propagating signals and transmission media.As used herein, “tangible computer readable storage medium” and“tangible machine-readable storage medium” are used interchangeably.Additionally or alternatively, the example processes of FIGS. 4, 6, 7,10-14, 16 and 17 may be implemented using coded instructions (e.g.,computer and/or machine-readable instructions) stored on anon-transitory computer and/or machine-readable medium such as a harddisk drive, a flash memory, a read-only memory, a compact disk, adigital versatile disk, a cache, a random-access memory and/or any otherstorage device or storage disk in which information is stored for anyduration (e.g., for extended time periods, permanently, for briefinstances, for temporarily buffering, and/or for caching of theinformation). As used herein, the term non-transitory computer readablemedium is expressly defined to include any type of computer readablestorage device and/or storage disk and to exclude propagating signalsand transmission media. As used herein, when the phrase “at least” isused as the transition term in a preamble of a claim, it is open-endedin the same manner as the term “comprising” is open ended.

FIG. 24 is a block diagram of an example processor platform 2400 capableof executing the instructions to implement the compliance monitor 218 ofFIGS. 2 and/or 3, the compliance assessor 302 of FIGS. 3 and/or 5, thecompliance measurer 304 of FIGS. 3 and/or 9, and/or the resultsprioritizer 306 of FIGS. 3 and/or 15. The processor platform 2400 canbe, for example, a server or any other type of computing device.

The processor platform 2400 of the illustrated example includes aprocessor 2412. The processor 2412 of the illustrated example ishardware (e.g., a semi-conductor based logic circuit). For example, theprocessor 2412 can be implemented by one or more integrated circuits,logic circuits, microprocessors or controllers from any desired familyor manufacturer.

The processor 2412 of the illustrated example includes a local memory2413 (e.g., a cache). The processor 2412 of the illustrated example isin communication with a main memory including a volatile memory 2414 anda non-volatile memory 2416 via a bus 2418. The volatile memory 2414 maybe implemented by Synchronous Dynamic Random Access Memory (SDRAM),Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory(RDRAM) and/or any other type of random access memory device. Thenon-volatile memory 2416 may be implemented by flash memory and/or anyother desired type of memory device. Access to the main memory 2414,2416 is controlled by a memory controller.

The processor platform 2400 of the illustrated example also includes aninterface circuit 2420. The interface circuit 2420 may be implemented byany type of interface standard, such as an Ethernet interface, auniversal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices 2422 are connectedto the interface circuit 2420. The input device(s) 2422 permit(s) a userto enter data and commands into the processor 2412. The input device(s)can be implemented by, for example, an audio sensor, a microphone, acamera (still or video), a keyboard, a button, a mouse, a touchscreen, atrack-pad, a trackball, isopoint and/or a voice recognition system.

One or more output devices 2424 are also connected to the interfacecircuit 2420 of the illustrated example. The output devices 2424 can beimplemented, for example, by display devices (e.g., a light emittingdiode (LED), an organic light emitting diode (OLED), a liquid crystaldisplay, a cathode ray tube display (CRT), a touchscreen, a tactileoutput device, a light emitting diode (LED), a printer and/or speakers).The interface circuit 2420 of the illustrated example, thus, typicallyincludes a graphics driver card, a graphics driver chip or a graphicsdriver processor.

The interface circuit 2420 of the illustrated example also includes acommunication device such as a transmitter, a receiver, a transceiver, amodem and/or network interface card to facilitate exchange of data withexternal machines (e.g., computing devices of any kind) via a network2426 (e.g., an Ethernet connection, a digital subscriber line (DSL), atelephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform 2400 of the illustrated example also includes oneor more mass storage devices 2428 for storing software and/or data.Examples of such mass storage devices 2428 include floppy disk drives,hard drive disks, compact disk drives, Blu-ray disk drives, RAIDsystems, and digital versatile disk (DVD) drives.

The coded instructions 2432 of FIGS. 4, 6, 7, 10-14, 16 and/or 17 may bestored in the mass storage device 2428, in the volatile memory 2414, inthe non-volatile memory 2416, and/or on a removable tangible computerreadable storage medium such as a CD or DVD.

From the foregoing, it will be appreciated that above disclosed methods,apparatus and articles of manufacture enable determining compliance of acomputing environment in real-time by assessing compliance in responseto a generated event.

Although certain example methods, apparatus and articles of manufacturehave been disclosed herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods,apparatus and articles of manufacture fairly falling within the scope ofthe claims of this patent.

1-20. (canceled)
 21. A method, comprising: monitoring a feed for anoccurrence of a change in a configuration of a computing resource in acomputing environment; parsing the feed to identify the computingresource; identifying additional computing resources related to thecomputing resource; retrieving a compliance policy comprising aplurality of rules comprising a plurality of scope conditions whichspecify whether the plurality of rules apply to the computing resource;and in an instance in which the occurrence of the change in theconfiguration of the computing resource is detected, assessingcompliance of the additional computing resources with the compliancepolicy.
 22. The method of claim 21, wherein identifying additionalcomputing resources comprises: retrieving an inventory of resources froma virtualization manager that maintains a dynamic list of computingresources that relate to other computing resources.
 23. The method ofclaim 21, wherein a rule of the plurality of rules comprises a pluralityof check conditions, the method further comprising: assessing complianceof the computing resource with the compliance policy; and storing acompliance events record comprising an indication of compliance statesof the computing resource with respect to the plurality of checkconditions.
 24. The method of claim 21, wherein a rule of the pluralityof rules comprises a check condition that defines a Boolean-valuedlogical expression, the Boolean-valued logical expression comprising aconfiguration setting or a property of the computing resource and avalue for the configuration setting or the property.
 25. The method ofclaim 24, wherein the check condition further comprises a reference byname to another check condition of the rule or another rule of theplurality of rules.
 26. The method of claim 21, further comprising: inan instance in which it is determined that the computing resource failsto satisfy a scope condition of the plurality of scope conditions,testing at least one of the additional computing resources thatsatisfies the scope condition against the compliance policy.
 27. Themethod of claim 26, wherein testing the at least one of the additionalcomputing resources comprises: determining that the at least one of theadditional computing resources satisfies a check condition of a rule ofthe compliance policy indicative of the at least one of the additionalcomputing resources satisfying the rule.
 28. A system, comprising: acomputing device; an application executable by the computing device,wherein when executed the application causes the computing device to atleast: monitor a feed for an occurrence of a change in a configurationof a computing resource in a computing environment; parse the feed toidentify the computing resource; identify additional computing resourcesrelated to the computing resource; retrieve a compliance policycomprising a plurality of rules comprising a plurality of scopeconditions which specify whether the plurality of rules apply to thecomputing resource; and in an instance in which the occurrence of thechange in the configuration of the computing resource is detected,assess compliance of the additional computing resources with thecompliance policy.
 29. The system of claim 28, wherein identifyadditional computing resources comprises: retrieve an inventory ofresources from a virtualization manager that maintains a dynamic list ofcomputing resources that relate to other computing resources.
 30. Thesystem of claim 28, wherein a rule of the plurality of rules comprises aplurality of check conditions, and wherein when executed the applicationfurther causes the computing device to at least: assess compliance ofthe computing resource with the compliance policy; and store acompliance events record comprising an indication of compliance statesof the computing resource with respect to the plurality of checkconditions.
 31. The system of claim 28, wherein a rule of the pluralityof rules comprises a check condition that defines a Boolean-valuedlogical expression, the Boolean-valued logical expression comprising aconfiguration setting or a property of the computing resource and avalue for the configuration setting or the property.
 32. The system ofclaim 31, wherein the check condition further comprises a reference byname to another check condition of the rule or another rule of theplurality of rules.
 33. The system of claim 28, wherein when executedthe application further causes the computing device to at least: in aninstance in which it is determined that the computing device fails tosatisfy a scope condition of the plurality of scope conditions, test atleast one of the additional computing resources that satisfies the scopecondition against the compliance policy.
 34. The system of claim 28,wherein test the at least one of the additional computing resourcescomprises: determine that the at least one of the additional computingresources satisfies a check condition of a rule of the compliance policyindicative of the at least one of the additional computing resourcessatisfying the rule.
 35. A non-transitory computer readable storagemedium comprising instructions that, when executed, cause at least oneprocessor to: monitor a feed for an occurrence of a change in aconfiguration of a computing resource in a computing environment; parsethe feed to identify the computing resource; identify additionalcomputing resources related to the computing resource; retrieve acompliance policy comprising a plurality of rules comprising a pluralityof scope conditions which specify whether the plurality of rules applyto the computing resource; and in an instance in which the occurrence ofthe change in the configuration of the computing resource is detected,assess compliance of the additional computing resources with thecompliance policy.
 36. The non-transitory computer readable storagemedium of claim 35, wherein identify additional computing resourcescomprises: retrieve an inventory of resources from a virtualizationmanager that maintains a dynamic list of computing resources that relateto other computing resources.
 37. The non-transitory computer readablestorage medium of claim 35, wherein a rule of the plurality of rulescomprises a plurality of check conditions, and wherein, when executed,the instructions further cause the at least one processor to: assesscompliance of the computing resource with the compliance policy; andstore a compliance events record comprising an indication of compliancestates of the computing resource with respect to the plurality of checkconditions.
 38. The non-transitory computer readable storage medium ofclaim 35, wherein a rule of the plurality of rules comprises a checkcondition that defines a Boolean-valued logical expression, theBoolean-valued logical expression comprising a configuration setting ora property of the computing resource and a value for the configurationsetting or the property.
 39. The non-transitory computer readablestorage medium of claim 35, wherein, when executed, the instructionsfurther cause the at least one processor to: in an instance in which itis determined that the computing resource fails to satisfy a scopecondition of the plurality of scope conditions, test at least one of theadditional computing resources that satisfies the scope conditionagainst the compliance policy.
 40. The non-transitory computer readablestorage medium of claim 39, wherein test the at least one of theadditional computing resources comprises: determine that the at leastone of the additional computing resources satisfies a check condition ofa rule of the compliance policy indicative of the at least one of theadditional computing resources satisfying the rule.